2024-03-25T16:03:12.762 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:03:12.763 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/orig.config.yaml 2024-03-25T16:03:12.821 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:03:12.907 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621243' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpd+Wy2Iyel6QblPc3uH35qdQGkFJ3LR0wpRQxPwEiG4A/yQjQmul8p7bjNIlRnJ5eUtXz3mw7mevtg6b9gLDA= smithi159.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNoDZvHzI5Yx7J7zf/elHmBDSn0RqC4PyojPaeAHlb5s3UwoIlI+7Q97xPwEovjZNdacoFeMV9QOEgynzu+UpP4= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:03:12.907 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:03:12.908 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:03:12.908 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:03:12.910 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:03:12.937 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:03:12.937 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:03:12.937 INFO:teuthology.packaging:ref: None 2024-03-25T16:03:12.937 INFO:teuthology.packaging:tag: None 2024-03-25T16:03:12.937 INFO:teuthology.packaging:branch: main 2024-03-25T16:03:12.937 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:03:12.937 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:03:13.302 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:03:13.304 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:03:13.313 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:03:13.313 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:03:13.326 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:03:13.344 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:03:13.354 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:03:13.378 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.840247', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpd+Wy2Iyel6QblPc3uH35qdQGkFJ3LR0wpRQxPwEiG4A/yQjQmul8p7bjNIlRnJ5eUtXz3mw7mevtg6b9gLDA='} 2024-03-25T16:03:13.400 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.838679', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNoDZvHzI5Yx7J7zf/elHmBDSn0RqC4PyojPaeAHlb5s3UwoIlI+7Q97xPwEovjZNdacoFeMV9QOEgynzu+UpP4='} 2024-03-25T16:03:13.400 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:03:13.411 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:03:13.411 INFO:teuthology.task.internal:roles: ubuntu@smithi159.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:03:13.411 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:03:13.491 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f270c5b80d0>, signals=[15]) 2024-03-25T16:03:13.491 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:03:13.502 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:03:13.502 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T16:03:13.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:03:13.599 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T16:03:13.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:03:13.691 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:03:13.698 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-03-25T16:03:13.715 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-03-25T16:03:13.716 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-03-25T16:03:13.773 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="CentOS Stream" 2024-03-25T16:03:13.872 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="9" 2024-03-25T16:03:13.872 INFO:teuthology.orchestra.run.smithi134.stdout:ID="centos" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="9" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:ANSI_COLOR="0;31" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:03:13.873 INFO:teuthology.orchestra.run.smithi134.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:03:13.874 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-03-25T16:03:13.902 DEBUG:teuthology.orchestra.run.smithi159:> uname -m 2024-03-25T16:03:13.919 INFO:teuthology.orchestra.run.smithi159.stdout:x86_64 2024-03-25T16:03:13.919 DEBUG:teuthology.orchestra.run.smithi159:> cat /etc/os-release 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:NAME="CentOS Stream" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION="9" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:ID="centos" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:VERSION_ID="9" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:ANSI_COLOR="0;31" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:03:13.976 INFO:teuthology.orchestra.run.smithi159.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:03:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:03:13.977 INFO:teuthology.orchestra.run.smithi159.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:03:13.977 INFO:teuthology.lock.ops:Updating smithi159.front.sepia.ceph.com on lock server 2024-03-25T16:03:14.002 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:03:14.012 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:03:14.068 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:03:14.068 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:03:14.071 DEBUG:teuthology.orchestra.run.smithi159:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:03:14.087 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:03:14.094 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:03:14.094 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:03:14.128 DEBUG:teuthology.orchestra.run.smithi159:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:03:14.292 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:03:14.343 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:03:14.361 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:03:14.361 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:03:14.362 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:03:14.362 DEBUG:teuthology.orchestra.run.smithi159:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:03:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:03:14.379 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-03-25T16:03:14.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:03:14.380 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-03-25T16:03:14.436 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:03:14.436 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.14.0-361.el9.x86_64 2024-03-25T16:03:14.436 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum install -y kernel 2024-03-25T16:03:14.438 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:03:14.438 INFO:teuthology.task.kernel:Running kernel on smithi159: 5.14.0-361.el9.x86_64 2024-03-25T16:03:14.438 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum install -y kernel 2024-03-25T16:03:16.679 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T16:03:16.711 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:03:18.354 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-25T16:03:18.442 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-25T16:03:19.111 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-25T16:03:19.843 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 7.2 MB/s | 8.0 MB 00:01 2024-03-25T16:03:21.262 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-25T16:03:22.571 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-25T16:03:25.305 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.1 MB 00:00 2024-03-25T16:03:26.547 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-25T16:03:27.688 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T16:03:28.205 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-03-25T16:03:30.159 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 21 MB 00:02 2024-03-25T16:03:31.763 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 6.0 MB/s | 21 MB 00:03 2024-03-25T16:03:35.181 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:03:36.688 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-25T16:03:37.296 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:03:37.296 INFO:teuthology.orchestra.run.smithi134.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:03:37.400 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:03:37.408 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:Install 4 Packages 2024-03-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 76 M 2024-03-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 123 M 2024-03-25T16:03:37.409 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:03:37.633 INFO:teuthology.orchestra.run.smithi134.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 681 kB/s | 44 kB 00:00 2024-03-25T16:03:38.737 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:03:38.737 INFO:teuthology.orchestra.run.smithi159.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:03:38.766 INFO:teuthology.orchestra.run.smithi134.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-03-25T16:03:38.830 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:03:38.833 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:03:38.834 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:03:38.834 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:03:38.834 INFO:teuthology.orchestra.run.smithi159.stdout:Install 4 Packages 2024-03-25T16:03:38.834 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:03:38.834 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 76 M 2024-03-25T16:03:38.834 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 123 M 2024-03-25T16:03:38.834 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:03:39.108 INFO:teuthology.orchestra.run.smithi134.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-25T16:03:39.240 INFO:teuthology.orchestra.run.smithi159.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 359 kB/s | 44 kB 00:00 2024-03-25T16:03:39.533 INFO:teuthology.orchestra.run.smithi134.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 17 MB/s | 33 MB 00:01 2024-03-25T16:03:39.533 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:03:39.534 INFO:teuthology.orchestra.run.smithi134.stdout:Total 36 MB/s | 76 MB 00:02 2024-03-25T16:03:40.203 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:03:40.553 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:03:40.670 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:03:41.098 INFO:teuthology.orchestra.run.smithi159.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.5 MB/s | 15 MB 00:01 2024-03-25T16:03:41.550 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:03:41.551 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:03:42.390 INFO:teuthology.orchestra.run.smithi159.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 8.7 MB/s | 27 MB 00:03 2024-03-25T16:03:44.027 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:03:44.431 INFO:teuthology.orchestra.run.smithi159.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 6.3 MB/s | 33 MB 00:05 2024-03-25T16:03:44.432 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:03:44.432 INFO:teuthology.orchestra.run.smithi159.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-25T16:03:44.721 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:03:44.808 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:45.094 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:03:45.476 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:03:45.477 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:03:46.493 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:03:46.493 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:03:47.549 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:47.686 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:48.978 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:03:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:03:49.790 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:52.387 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:52.486 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:52.556 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:03:52.692 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:57.202 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:03:57.414 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:03:57.502 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:02.247 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:26.387 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:26.387 INFO:teuthology.orchestra.run.smithi134.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:04:26.387 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:04:26.431 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:26.809 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:26.810 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:26.810 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:26.810 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:04:27.203 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:04:27.547 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:27.650 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:27.651 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:27.651 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:04:27.651 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:04:27.651 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:04:27.651 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:04:27.651 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:04:27.651 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:27.731 INFO:teuthology.orchestra.run.smithi134.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:27.732 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:27.732 DEBUG:teuthology.orchestra.run.smithi134:> sudo rpm -qi grub2-tools 2024-03-25T16:04:27.776 INFO:teuthology.orchestra.run.smithi134.stdout:Name : grub2-tools 2024-03-25T16:04:27.776 INFO:teuthology.orchestra.run.smithi134.stdout:Epoch : 1 2024-03-25T16:04:27.776 INFO:teuthology.orchestra.run.smithi134.stdout:Version : 2.06 2024-03-25T16:04:27.776 INFO:teuthology.orchestra.run.smithi134.stdout:Release : 68.el9 2024-03-25T16:04:27.776 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: x86_64 2024-03-25T16:04:27.776 INFO:teuthology.orchestra.run.smithi134.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:04:27.776 INFO:teuthology.orchestra.run.smithi134.stdout:Group : Unspecified 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Size : 8264254 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:License : GPLv3+ 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Packager : builder@centos.org 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Vendor : CentOS 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Summary : Support tools for GRUB. 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:Description : 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:04:27.777 INFO:teuthology.orchestra.run.smithi134.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:04:27.778 INFO:teuthology.orchestra.run.smithi134.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:04:27.778 INFO:teuthology.orchestra.run.smithi134.stdout:hardware devices. 2024-03-25T16:04:27.778 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:04:27.778 INFO:teuthology.orchestra.run.smithi134.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:04:27.779 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:04:27.779 INFO:teuthology.task.kernel:Updating grub on smithi134 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:04:27.779 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:04:28.341 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-03-25T16:04:29.840 INFO:teuthology.orchestra.run.smithi134.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:04:29.860 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-03-25T16:04:29.863 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-25T16:04:29.878 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.87NHeIHUmU 2024-03-25T16:04:29.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.87NHeIHUmU 2024-03-25T16:04:29.959 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.87NHeIHUmU 2024-03-25T16:04:30.105 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.87NHeIHUmU is 6KB 2024-03-25T16:04:30.155 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.87NHeIHUmU 2024-03-25T16:04:30.169 DEBUG:teuthology.orchestra.run.smithi134:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:04:30.233 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:04:30.234 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:04:30.234 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:04:30.234 INFO:teuthology.orchestra.run.smithi134.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:04:30.235 DEBUG:teuthology.orchestra.run.smithi134:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:04:30.374 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-03-25T16:04:31.520 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:31.520 INFO:teuthology.orchestra.run.smithi159.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:04:31.521 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:04:31.564 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:31.950 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:04:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:04:31.982 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:04:32.337 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:04:32.338 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:04:32.659 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:32.756 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:32.756 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:32.756 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:04:32.757 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:04:32.757 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:04:32.757 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:04:32.757 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi159.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:04:32.757 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:04:32.837 INFO:teuthology.orchestra.run.smithi159.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:04:32.838 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:04:32.838 DEBUG:teuthology.orchestra.run.smithi159:> sudo rpm -qi grub2-tools 2024-03-25T16:04:32.872 INFO:teuthology.orchestra.run.smithi159.stdout:Name : grub2-tools 2024-03-25T16:04:32.872 INFO:teuthology.orchestra.run.smithi159.stdout:Epoch : 1 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Version : 2.06 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Release : 68.el9 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Architecture: x86_64 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Group : Unspecified 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Size : 8264254 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:License : GPLv3+ 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Packager : builder@centos.org 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:Vendor : CentOS 2024-03-25T16:04:32.873 INFO:teuthology.orchestra.run.smithi159.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout:Summary : Support tools for GRUB. 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout:Description : 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout:hardware devices. 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:04:32.874 INFO:teuthology.orchestra.run.smithi159.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:04:32.875 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:04:32.875 INFO:teuthology.task.kernel:Updating grub on smithi159 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:04:32.875 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:04:33.492 INFO:teuthology.orchestra.run.smithi159.stderr:Generating grub configuration file ... 2024-03-25T16:04:34.885 INFO:teuthology.orchestra.run.smithi159.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:04:34.915 INFO:teuthology.orchestra.run.smithi159.stderr:done 2024-03-25T16:04:34.917 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-25T16:04:34.935 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.mjtpqoVX2s 2024-03-25T16:04:34.935 DEBUG:teuthology.orchestra.run.smithi159:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mjtpqoVX2s 2024-03-25T16:04:35.004 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.mjtpqoVX2s 2024-03-25T16:04:35.145 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.mjtpqoVX2s is 6KB 2024-03-25T16:04:35.197 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.mjtpqoVX2s 2024-03-25T16:04:35.211 DEBUG:teuthology.orchestra.run.smithi159:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:04:35.280 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:04:35.280 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:04:35.281 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:04:35.281 INFO:teuthology.orchestra.run.smithi159.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:04:35.281 DEBUG:teuthology.orchestra.run.smithi159:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:04:35.417 DEBUG:teuthology.orchestra.run.smithi159:> sudo shutdown -r now 2024-03-25T16:05:00.398 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:05:00.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T16:05:00.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:05:05.422 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:05:05.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T16:05:05.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:05:39.999 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-03-25T16:05:49.022 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T16:05:49.023 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:00.411 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:06:07.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-03-25T16:06:09.414 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T16:06:09.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:09.835 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-03-25T16:06:10.292 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T16:06:10.293 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:06:10.293 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-03-25T16:06:10.307 INFO:teuthology.orchestra.run.smithi134.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:06:10.307 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:06:10.307 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:06:10.307 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:06:11.309 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:06:11.309 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:06:11.396 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-03-25T16:06:11.427 DEBUG:teuthology.parallel:result is None 2024-03-25T16:06:19.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T16:06:19.392 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:22.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.159 2024-03-25T16:06:37.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T16:06:37.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:06:37.871 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-03-25T16:06:38.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T16:06:38.239 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:06:38.239 DEBUG:teuthology.orchestra.run.smithi159:> uname -r 2024-03-25T16:06:38.254 INFO:teuthology.orchestra.run.smithi159.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:06:38.254 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:06:38.254 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:06:38.254 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:06:39.255 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:06:39.255 DEBUG:teuthology.orchestra.run.smithi159:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:06:39.343 INFO:teuthology.orchestra.run.smithi159.stdout:ttyS1 2024-03-25T16:06:39.368 DEBUG:teuthology.parallel:result is None 2024-03-25T16:06:39.369 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:06:39.378 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:06:39.378 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:06:39.381 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:06:39.398 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:06:39.409 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:06:39.423 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:06:39.423 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:06:39.439 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:06:39.543 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:06:39.554 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:06:39.554 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:06:39.557 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:06:39.596 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:39.596 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:39.610 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:39.610 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:06:39.612 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:06:39.620 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:06:39.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:06:39.654 DEBUG:teuthology.orchestra.run.smithi159:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:06:39.683 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:06:39.692 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:06:39.692 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:06:39.724 DEBUG:teuthology.orchestra.run.smithi159:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:06:39.740 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:39.822 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:39.869 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:39.951 DEBUG:teuthology.orchestra.run.smithi134:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:39.995 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T16:06:39.995 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:06:40.062 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:40.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:06:40.155 DEBUG:teuthology.orchestra.run.smithi159:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:40.234 DEBUG:teuthology.orchestra.run.smithi159:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:06:40.282 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T16:06:40.282 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:06:40.350 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-03-25T16:06:40.355 DEBUG:teuthology.orchestra.run.smithi159:> sudo service rsyslog restart 2024-03-25T16:06:40.392 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:06:40.438 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:06:40.819 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:06:40.827 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:06:40.827 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:06:40.883 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:06:40.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo service auditd rotate 2024-03-25T16:06:40.952 INFO:teuthology.orchestra.run.smithi134.stdout:Rotating logs: 2024-03-25T16:06:40.954 DEBUG:teuthology.orchestra.run.smithi159:> sudo service auditd rotate 2024-03-25T16:06:41.038 INFO:teuthology.orchestra.run.smithi159.stdout:Rotating logs: 2024-03-25T16:06:41.039 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:06:41.040 DEBUG:teuthology.orchestra.run.smithi134:> /usr/sbin/getenforce 2024-03-25T16:06:41.057 INFO:teuthology.orchestra.run.smithi134.stdout:Permissive 2024-03-25T16:06:41.057 DEBUG:teuthology.orchestra.run.smithi159:> /usr/sbin/getenforce 2024-03-25T16:06:41.084 INFO:teuthology.orchestra.run.smithi159.stdout:Permissive 2024-03-25T16:06:41.085 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi134.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi159.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:06:41.085 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:06:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:06:41.125 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:06:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:06:41.154 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:06:41.154 DEBUG:teuthology.orchestra.run.smithi134:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:06:41.201 DEBUG:teuthology.orchestra.run.smithi159:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:06:41.237 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:06:41.247 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:06:41.264 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:06:41.295 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com,smithi159.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:11:25.371 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com'), Remote(name='ubuntu@smithi159.front.sepia.ceph.com')] 2024-03-25T16:11:25.392 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T16:11:25.393 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:25.485 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-03-25T16:11:25.518 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-03-25T16:11:25.519 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T16:11:25.519 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi159.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:11:25.609 DEBUG:teuthology.orchestra.run.smithi159:> true 2024-03-25T16:11:25.689 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi159.front.sepia.ceph.com' 2024-03-25T16:11:25.690 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:11:25.701 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:11:25.701 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:11:25.702 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:11:25.704 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:11:25.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:11:25.732 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:11:25.748 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:11:25.760 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:11:25.777 INFO:teuthology.orchestra.run.smithi134.stderr:sudo: ntpd: command not found 2024-03-25T16:11:25.777 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:11:25.790 INFO:teuthology.orchestra.run.smithi134.stdout:506 Cannot talk to daemon 2024-03-25T16:11:25.806 INFO:teuthology.orchestra.run.smithi159.stderr:sudo: ntpd: command not found 2024-03-25T16:11:25.807 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:11:25.819 INFO:teuthology.orchestra.run.smithi159.stdout:506 Cannot talk to daemon 2024-03-25T16:11:25.823 INFO:teuthology.orchestra.run.smithi134.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:11:25.836 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:11:25.852 INFO:teuthology.orchestra.run.smithi159.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:11:25.874 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:11:25.878 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:11:25.878 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-25T16:11:25.878 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.878 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.878 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.878 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.904 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:11:25.907 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:11:25.907 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-25T16:11:25.907 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.907 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.907 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.908 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:11:25.908 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:11:25.917 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:11:25.917 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:11:25.917 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:11:25.917 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:11:25.923 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:11:25.923 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:11:25.923 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:11:25.923 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:11:25.925 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:11:25.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:11:26.071 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:11:26.071 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:11:26.183 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:11:26.183 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T16:11:26.183 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:11:26.187 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:11:26.187 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:11:26.211 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:11:26.211 DEBUG:teuthology.orchestra.run.smithi159:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:11:26.279 DEBUG:teuthology.orchestra.run.smithi159:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:11:26.295 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:11:26.295 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T16:11:26.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:11:26.324 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:11:26.324 DEBUG:teuthology.orchestra.run.smithi134:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:11:26.359 DEBUG:teuthology.orchestra.run.smithi159:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:11:26.392 DEBUG:teuthology.orchestra.run.smithi134:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:11:26.426 INFO:teuthology.orchestra.run.smithi159.stdout:check_obsoletes = 1 2024-03-25T16:11:26.427 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum clean all 2024-03-25T16:11:26.472 DEBUG:teuthology.orchestra.run.smithi134:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:11:26.537 INFO:teuthology.orchestra.run.smithi134.stdout:check_obsoletes = 1 2024-03-25T16:11:26.539 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum clean all 2024-03-25T16:11:26.664 INFO:teuthology.orchestra.run.smithi159.stdout:74 files removed 2024-03-25T16:11:26.692 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw 2024-03-25T16:11:26.772 INFO:teuthology.orchestra.run.smithi134.stdout:74 files removed 2024-03-25T16:11:26.799 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw 2024-03-25T16:11:27.423 INFO:teuthology.orchestra.run.smithi134.stdout:ceph packages for x86_64 253 kB/s | 82 kB 00:00 2024-03-25T16:11:27.651 INFO:teuthology.orchestra.run.smithi159.stdout:ceph packages for x86_64 125 kB/s | 82 kB 00:00 2024-03-25T16:11:27.907 INFO:teuthology.orchestra.run.smithi134.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-25T16:11:27.960 INFO:teuthology.orchestra.run.smithi159.stdout:ceph noarch packages 39 kB/s | 10 kB 00:00 2024-03-25T16:11:28.533 INFO:teuthology.orchestra.run.smithi134.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-25T16:11:28.610 INFO:teuthology.orchestra.run.smithi159.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2024-03-25T16:11:29.452 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - AppStream 7.6 MB/s | 6.7 MB 00:00 2024-03-25T16:11:29.496 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-03-25T16:11:31.036 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS-9 - BaseOS 4.1 MB/s | 2.0 MB 00:00 2024-03-25T16:11:31.180 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-25T16:11:32.236 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - BaseOS 9.2 MB/s | 8.0 MB 00:00 2024-03-25T16:11:32.646 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.0 MB 00:01 2024-03-25T16:11:35.228 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - AppStream 9.6 MB/s | 19 MB 00:01 2024-03-25T16:11:35.588 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 19 MB 00:01 2024-03-25T16:11:39.262 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:11:39.564 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T16:11:40.787 INFO:teuthology.orchestra.run.smithi134.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-25T16:11:41.167 INFO:teuthology.orchestra.run.smithi159.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 16 kB 00:00 2024-03-25T16:11:42.729 INFO:teuthology.orchestra.run.smithi159.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-25T16:11:43.214 INFO:teuthology.orchestra.run.smithi134.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-25T16:11:47.760 INFO:teuthology.orchestra.run.smithi159.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:11:48.339 INFO:teuthology.orchestra.run.smithi134.stdout:lab-extras 5.9 kB/s | 1.7 kB 00:00 2024-03-25T16:11:50.229 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrading: 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:11:50.259 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:11:50.260 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:11:50.261 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout:Installing weak dependencies: 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout:Install 30 Packages 2024-03-25T16:11:50.262 INFO:teuthology.orchestra.run.smithi159.stdout:Upgrade 2 Packages 2024-03-25T16:11:50.263 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:11:50.263 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 58 M 2024-03-25T16:11:50.263 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:11:50.768 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrading: 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:11:50.780 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:11:50.781 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:11:50.782 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout:Installing weak dependencies: 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:11:50.783 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:11:50.784 INFO:teuthology.orchestra.run.smithi134.stdout:Install 30 Packages 2024-03-25T16:11:50.784 INFO:teuthology.orchestra.run.smithi134.stdout:Upgrade 2 Packages 2024-03-25T16:11:50.784 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:11:50.784 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 58 M 2024-03-25T16:11:50.784 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:11:54.994 INFO:teuthology.orchestra.run.smithi159.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 5.3 MB 00:04 2024-03-25T16:11:55.094 INFO:teuthology.orchestra.run.smithi159.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:11:55.494 INFO:teuthology.orchestra.run.smithi159.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.8 MB/s | 742 kB 00:00 2024-03-25T16:11:55.786 INFO:teuthology.orchestra.run.smithi159.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.7 MB/s | 498 kB 00:00 2024-03-25T16:11:56.481 INFO:teuthology.orchestra.run.smithi134.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 3.6 MB/s | 20 MB 00:05 2024-03-25T16:11:56.540 INFO:teuthology.orchestra.run.smithi134.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 434 kB/s | 25 kB 00:00 2024-03-25T16:11:56.715 INFO:teuthology.orchestra.run.smithi134.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.1 MB/s | 742 kB 00:00 2024-03-25T16:11:56.840 INFO:teuthology.orchestra.run.smithi134.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 3.9 MB/s | 498 kB 00:00 2024-03-25T16:11:57.866 INFO:teuthology.orchestra.run.smithi134.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 5.0 MB/s | 5.1 MB 00:01 2024-03-25T16:11:57.949 INFO:teuthology.orchestra.run.smithi134.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 541 kB/s | 45 kB 00:00 2024-03-25T16:11:58.036 INFO:teuthology.orchestra.run.smithi159.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 9.7 MB 00:07 2024-03-25T16:11:58.058 INFO:teuthology.orchestra.run.smithi134.stdout:(7/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 9.7 MB 00:07 2024-03-25T16:11:58.083 INFO:teuthology.orchestra.run.smithi134.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 982 kB/s | 131 kB 00:00 2024-03-25T16:11:58.103 INFO:teuthology.orchestra.run.smithi159.stdout:(6/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 2.2 MB/s | 5.1 MB 00:02 2024-03-25T16:11:58.120 INFO:teuthology.orchestra.run.smithi159.stdout:(7/32): python3-ceph-argparse-19.0.0-1439.g5dcb 541 kB/s | 45 kB 00:00 2024-03-25T16:11:58.175 INFO:teuthology.orchestra.run.smithi134.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 320 kB 00:00 2024-03-25T16:11:58.201 INFO:teuthology.orchestra.run.smithi134.stdout:(10/32): python3-cephfs-19.0.0-1439.g5dcb190d.e 1.1 MB/s | 162 kB 00:00 2024-03-25T16:11:58.237 INFO:teuthology.orchestra.run.smithi159.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 983 kB/s | 131 kB 00:00 2024-03-25T16:11:58.267 INFO:teuthology.orchestra.run.smithi134.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.2 MB/s | 298 kB 00:00 2024-03-25T16:11:58.279 INFO:teuthology.orchestra.run.smithi159.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.0 MB/s | 162 kB 00:00 2024-03-25T16:11:58.284 INFO:teuthology.orchestra.run.smithi134.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-25T16:11:58.454 INFO:teuthology.orchestra.run.smithi159.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 320 kB 00:00 2024-03-25T16:11:58.488 INFO:teuthology.orchestra.run.smithi159.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 298 kB 00:00 2024-03-25T16:11:58.501 INFO:teuthology.orchestra.run.smithi134.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 454 kB/s | 106 kB 00:00 2024-03-25T16:11:58.526 INFO:teuthology.orchestra.run.smithi134.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 658 kB/s | 159 kB 00:00 2024-03-25T16:11:58.569 INFO:teuthology.orchestra.run.smithi134.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 660 kB/s | 44 kB 00:00 2024-03-25T16:11:58.580 INFO:teuthology.orchestra.run.smithi159.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 793 kB/s | 99 kB 00:00 2024-03-25T16:11:58.744 INFO:teuthology.orchestra.run.smithi134.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.0 MB/s | 661 kB 00:00 2024-03-25T16:11:58.770 INFO:teuthology.orchestra.run.smithi134.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-03-25T16:11:58.797 INFO:teuthology.orchestra.run.smithi159.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 343 kB/s | 106 kB 00:00 2024-03-25T16:11:58.811 INFO:teuthology.orchestra.run.smithi134.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 618 kB/s | 41 kB 00:00 2024-03-25T16:11:58.872 INFO:teuthology.orchestra.run.smithi159.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 591 kB/s | 44 kB 00:00 2024-03-25T16:11:58.895 INFO:teuthology.orchestra.run.smithi134.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 418 kB/s | 35 kB 00:00 2024-03-25T16:11:58.905 INFO:teuthology.orchestra.run.smithi159.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 488 kB/s | 159 kB 00:00 2024-03-25T16:11:58.920 INFO:teuthology.orchestra.run.smithi134.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.4 MB/s | 216 kB 00:00 2024-03-25T16:11:58.954 INFO:teuthology.orchestra.run.smithi134.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-03-25T16:11:59.073 INFO:teuthology.orchestra.run.smithi159.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 661 kB 00:00 2024-03-25T16:11:59.098 INFO:teuthology.orchestra.run.smithi159.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 282 kB 00:00 2024-03-25T16:11:59.132 INFO:teuthology.orchestra.run.smithi159.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 712 kB/s | 41 kB 00:00 2024-03-25T16:11:59.173 INFO:teuthology.orchestra.run.smithi159.stdout:(19/32): qatlib-23.11.0-1.el9.x86_64.rpm 2.8 MB/s | 216 kB 00:00 2024-03-25T16:11:59.190 INFO:teuthology.orchestra.run.smithi159.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 596 kB/s | 35 kB 00:00 2024-03-25T16:11:59.233 INFO:teuthology.orchestra.run.smithi159.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 759 kB/s | 44 kB 00:00 2024-03-25T16:11:59.599 INFO:teuthology.orchestra.run.smithi159.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 756 kB/s | 309 kB 00:00 2024-03-25T16:11:59.721 INFO:teuthology.orchestra.run.smithi134.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 386 kB/s | 309 kB 00:00 2024-03-25T16:11:59.733 INFO:teuthology.orchestra.run.smithi159.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 194 kB/s | 26 kB 00:00 2024-03-25T16:11:59.829 INFO:teuthology.orchestra.run.smithi134.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-25T16:11:59.858 INFO:teuthology.orchestra.run.smithi159.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-03-25T16:11:59.980 INFO:teuthology.orchestra.run.smithi134.stdout:(24/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x8 609 kB/s | 5.3 MB 00:08 2024-03-25T16:11:59.997 INFO:teuthology.orchestra.run.smithi134.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 291 kB/s | 49 kB 00:00 2024-03-25T16:12:00.101 INFO:teuthology.orchestra.run.smithi159.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 279 kB/s | 67 kB 00:00 2024-03-25T16:12:00.130 INFO:teuthology.orchestra.run.smithi134.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2024-03-25T16:12:00.280 INFO:teuthology.orchestra.run.smithi134.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-25T16:12:00.306 INFO:teuthology.orchestra.run.smithi134.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-03-25T16:12:00.448 INFO:teuthology.orchestra.run.smithi134.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-03-25T16:12:00.542 INFO:teuthology.orchestra.run.smithi159.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.9 MB/s | 838 kB 00:00 2024-03-25T16:12:00.598 INFO:teuthology.orchestra.run.smithi134.stdout:(30/32): libunwind-1.6.2-1.el9.x86_64.rpm 109 kB/s | 67 kB 00:00 2024-03-25T16:12:00.651 INFO:teuthology.orchestra.run.smithi159.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2024-03-25T16:12:00.984 INFO:teuthology.orchestra.run.smithi159.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 434 kB/s | 192 kB 00:00 2024-03-25T16:12:01.101 INFO:teuthology.orchestra.run.smithi159.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2024-03-25T16:12:01.356 INFO:teuthology.orchestra.run.smithi134.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.4 MB/s | 3.1 MB 00:00 2024-03-25T16:12:01.968 INFO:teuthology.orchestra.run.smithi159.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 3.1 MB 00:00 2024-03-25T16:12:02.410 INFO:teuthology.orchestra.run.smithi159.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.4 MB/s | 3.4 MB 00:01 2024-03-25T16:12:03.715 INFO:teuthology.orchestra.run.smithi134.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 3.4 MB 00:03 2024-03-25T16:12:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:03.743 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.5 MB/s | 58 MB 00:12 2024-03-25T16:12:03.756 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:12:03.795 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:12:03.795 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:12:04.272 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:12:04.272 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:12:05.726 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:12:05.810 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:12:05.845 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:12:05.870 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:12:05.901 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:05.918 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:05.976 INFO:teuthology.orchestra.run.smithi159.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.3 MB/s | 20 MB 00:15 2024-03-25T16:12:05.981 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:05.981 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.7 MB/s | 58 MB 00:15 2024-03-25T16:12:06.016 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:12:06.054 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:12:06.054 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:12:06.101 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:06.156 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:06.483 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:12:06.513 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:12:06.514 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:12:06.539 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:06.650 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:06.734 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:12:06.765 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:06.828 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:06.856 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:12:06.900 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:12:06.968 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:06.992 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:07.038 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:07.066 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:12:07.172 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:12:07.201 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:12:07.241 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:12:07.274 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:12:07.303 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:12:07.335 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:12:07.653 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:12:07.691 INFO:teuthology.orchestra.run.smithi134.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:07.780 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:07.812 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:12:07.850 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:12:07.973 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:12:08.114 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:12:08.148 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:12:08.178 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:12:08.213 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:08.230 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:12:08.318 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:12:08.343 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:08.397 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:08.431 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:08.481 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:12:08.485 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:12:08.512 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:12:08.808 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:12:08.858 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:08.934 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:09.022 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:12:09.051 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:09.106 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:12:09.144 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:12:09.196 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:12:09.303 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:09.329 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:09.393 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:12:09.439 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:12:09.560 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:12:09.595 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:12:09.642 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:12:09.677 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:12:09.705 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:12:09.738 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:12:10.063 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:12:10.093 INFO:teuthology.orchestra.run.smithi159.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:10.166 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:12:10.199 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:12:10.248 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:12:10.675 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:10.698 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:12:10.709 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:10.729 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:10.792 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:12:10.869 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:12:10.899 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:12:11.273 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:11.329 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:11.419 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:11.419 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:12:11.419 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:12:11.420 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:11.447 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:13.086 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:13.128 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:13.721 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:12:13.765 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:13.862 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:12:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:12:13.862 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:12:13.862 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:13.908 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:20.825 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:20.825 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /sys 2024-03-25T16:12:20.825 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /proc 2024-03-25T16:12:20.825 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /mnt 2024-03-25T16:12:20.825 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /var/tmp 2024-03-25T16:12:20.826 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /home 2024-03-25T16:12:20.826 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /root 2024-03-25T16:12:20.826 INFO:teuthology.orchestra.run.smithi134.stdout:skipping the directory /tmp 2024-03-25T16:12:20.826 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:21.733 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:21.763 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:21.763 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:12:21.763 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:21.763 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:21.763 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:21.765 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:21.836 INFO:teuthology.orchestra.run.smithi134.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:12:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:12:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:12:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:12:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:12:22.929 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:12:22.930 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:22.931 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:12:22.932 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:12:23.057 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:23.057 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /sys 2024-03-25T16:12:23.057 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /proc 2024-03-25T16:12:23.058 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /mnt 2024-03-25T16:12:23.058 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /var/tmp 2024-03-25T16:12:23.058 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /home 2024-03-25T16:12:23.058 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /root 2024-03-25T16:12:23.058 INFO:teuthology.orchestra.run.smithi159.stdout:skipping the directory /tmp 2024-03-25T16:12:23.058 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:23.966 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:23.991 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:12:23.991 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:23.991 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:12:23.991 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:23.991 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:12:23.991 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:23.993 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:24.060 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:12:24.060 INFO:teuthology.orchestra.run.smithi159.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout:Upgraded: 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.157 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:12:24.158 INFO:teuthology.orchestra.run.smithi134.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:12:24.159 INFO:teuthology.orchestra.run.smithi134.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:12:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:12:24.160 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:24.160 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:12:24.484 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test 2024-03-25T16:12:25.052 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:00:37 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:12:25.208 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:12:25.208 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:12:25.209 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:12:25.210 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:25.210 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:12:25.210 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:12:25.210 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-03-25T16:12:25.210 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:25.210 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-03-25T16:12:25.211 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 204 M 2024-03-25T16:12:25.211 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:12:25.235 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:12:25.236 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:12:25.237 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:12:25.238 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:12:25.238 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:12:25.239 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:12:25.239 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:12:25.239 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:12:25.239 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:12:25.570 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 687 kB/s | 217 kB 00:00 2024-03-25T16:12:25.803 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-25T16:12:25.903 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 626 kB/s | 63 kB 00:00 2024-03-25T16:12:26.345 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 150 kB/s | 163 kB 00:01 2024-03-25T16:12:26.374 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:12:26.374 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:26.374 INFO:teuthology.orchestra.run.smithi159.stdout:Upgraded: 2024-03-25T16:12:26.374 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.374 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.374 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:12:26.374 INFO:teuthology.orchestra.run.smithi159.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:12:26.375 INFO:teuthology.orchestra.run.smithi159.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:12:26.376 INFO:teuthology.orchestra.run.smithi159.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:12:26.377 INFO:teuthology.orchestra.run.smithi159.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:12:26.377 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:26.377 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:12:26.697 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test 2024-03-25T16:12:26.971 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 175 kB/s | 187 kB 00:01 2024-03-25T16:12:27.257 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:00:40 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:12:27.405 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:12:27.406 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:12:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:12:27.406 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:12:27.406 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:12:27.406 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2024-03-25T16:12:27.407 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:27.408 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 48 M 2024-03-25T16:12:27.408 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 204 M 2024-03-25T16:12:27.408 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:12:27.848 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 781 kB/s | 217 kB 00:00 2024-03-25T16:12:27.949 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.9 MB/s | 300 kB 00:00 2024-03-25T16:12:27.999 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T16:12:28.124 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:12:28.449 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 186 kB/s | 163 kB 00:00 2024-03-25T16:12:40.057 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 3.8 MB/s | 47 MB 00:12 2024-03-25T16:12:40.060 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:40.060 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.8 MB/s | 48 MB 00:12 2024-03-25T16:12:40.074 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:12:40.097 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:12:40.097 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:12:40.478 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:12:40.478 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:12:41.060 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:12:41.132 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:12:41.172 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:12:41.213 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:12:41.260 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:12:41.285 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:12:45.771 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:12:45.844 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:12:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:12:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:12:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:12:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:12:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:12:48.263 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:48.635 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:12:48.932 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph 2024-03-25T16:12:48.985 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 47 MB 00:23 2024-03-25T16:12:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:48.988 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.0 MB/s | 48 MB 00:23 2024-03-25T16:12:49.003 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:12:49.024 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:12:49.024 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:12:49.408 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:12:49.409 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:12:49.507 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:02 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:12:49.663 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:12:49.665 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:12:49.666 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:12:49.667 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:12:49.668 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:12:49.669 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:49.669 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:12:49.669 INFO:teuthology.orchestra.run.smithi159.stdout:========================================================================================= 2024-03-25T16:12:49.669 INFO:teuthology.orchestra.run.smithi159.stdout:Install 36 Packages 2024-03-25T16:12:49.669 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:49.669 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 30 M 2024-03-25T16:12:49.670 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 107 M 2024-03-25T16:12:49.670 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:12:49.986 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:12:50.060 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:12:50.110 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:12:50.151 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:12:50.211 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:12:50.238 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:12:50.321 INFO:teuthology.orchestra.run.smithi159.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-03-25T16:12:50.729 INFO:teuthology.orchestra.run.smithi159.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.7 MB/s | 2.1 MB 00:00 2024-03-25T16:12:51.171 INFO:teuthology.orchestra.run.smithi159.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 5.5 MB/s | 4.7 MB 00:00 2024-03-25T16:12:51.238 INFO:teuthology.orchestra.run.smithi159.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-25T16:12:51.363 INFO:teuthology.orchestra.run.smithi159.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:12:51.405 INFO:teuthology.orchestra.run.smithi159.stdout:(6/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-25T16:12:51.430 INFO:teuthology.orchestra.run.smithi159.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-25T16:12:51.447 INFO:teuthology.orchestra.run.smithi159.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-25T16:12:51.464 INFO:teuthology.orchestra.run.smithi159.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:12:51.489 INFO:teuthology.orchestra.run.smithi159.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 925 kB/s | 39 kB 00:00 2024-03-25T16:12:51.515 INFO:teuthology.orchestra.run.smithi159.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-25T16:12:51.616 INFO:teuthology.orchestra.run.smithi159.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 9.1 MB/s | 1.1 MB 00:00 2024-03-25T16:12:51.680 INFO:teuthology.orchestra.run.smithi159.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 824 kB/s | 103 kB 00:00 2024-03-25T16:12:51.680 INFO:teuthology.orchestra.run.smithi159.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-25T16:12:51.691 INFO:teuthology.orchestra.run.smithi159.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-25T16:12:51.741 INFO:teuthology.orchestra.run.smithi159.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-03-25T16:12:51.808 INFO:teuthology.orchestra.run.smithi159.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 879 kB/s | 125 kB 00:00 2024-03-25T16:12:52.225 INFO:teuthology.orchestra.run.smithi159.stdout:(18/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 11 MB/s | 17 MB 00:01 2024-03-25T16:12:52.250 INFO:teuthology.orchestra.run.smithi159.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 104 kB/s | 43 kB 00:00 2024-03-25T16:12:52.334 INFO:teuthology.orchestra.run.smithi159.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 255 kB/s | 151 kB 00:00 2024-03-25T16:12:52.418 INFO:teuthology.orchestra.run.smithi159.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T16:12:52.502 INFO:teuthology.orchestra.run.smithi159.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 212 kB/s | 18 kB 00:00 2024-03-25T16:12:52.535 INFO:teuthology.orchestra.run.smithi159.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-03-25T16:12:52.585 INFO:teuthology.orchestra.run.smithi159.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 278 kB/s | 23 kB 00:00 2024-03-25T16:12:52.619 INFO:teuthology.orchestra.run.smithi159.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 233 kB/s | 19 kB 00:00 2024-03-25T16:12:52.636 INFO:teuthology.orchestra.run.smithi159.stdout:(26/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 419 kB/s | 172 kB 00:00 2024-03-25T16:12:52.669 INFO:teuthology.orchestra.run.smithi159.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 234 kB/s | 20 kB 00:00 2024-03-25T16:12:52.703 INFO:teuthology.orchestra.run.smithi159.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 551 kB/s | 46 kB 00:00 2024-03-25T16:12:52.728 INFO:teuthology.orchestra.run.smithi159.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 857 kB/s | 79 kB 00:00 2024-03-25T16:12:52.787 INFO:teuthology.orchestra.run.smithi159.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 197 kB/s | 16 kB 00:00 2024-03-25T16:12:52.829 INFO:teuthology.orchestra.run.smithi159.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 897 kB/s | 90 kB 00:00 2024-03-25T16:12:52.863 INFO:teuthology.orchestra.run.smithi159.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-25T16:12:52.888 INFO:teuthology.orchestra.run.smithi159.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 356 kB/s | 36 kB 00:00 2024-03-25T16:12:52.963 INFO:teuthology.orchestra.run.smithi159.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-25T16:12:52.980 INFO:teuthology.orchestra.run.smithi159.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-25T16:12:53.005 INFO:teuthology.orchestra.run.smithi159.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-25T16:12:53.012 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:12:53.012 INFO:teuthology.orchestra.run.smithi159.stdout:Total 8.9 MB/s | 30 MB 00:03 2024-03-25T16:12:53.038 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:12:53.080 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:12:53.080 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:12:53.524 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:12:53.525 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:12:54.686 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:12:54.773 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:12:54.863 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:12:54.924 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:12:54.966 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:12:54.993 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:12:55.064 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:12:55.119 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:12:55.158 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:12:55.250 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:12:55.336 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:12:55.386 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:12:55.458 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:12:55.512 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:12:55.550 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:12:55.598 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:12:55.654 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:12:55.842 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:12:55.888 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:12:55.926 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:12:55.961 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:12:55.991 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:12:56.059 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:12:56.093 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:12:56.126 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:12:56.167 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:12:56.202 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:12:56.236 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:12:56.327 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:12:56.376 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:12:56.449 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:12:56.592 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:12:56.711 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:12:56.871 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:12:56.897 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:12:56.923 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:12:56.923 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:56.923 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:12:56.923 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:12:56.923 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:12:56.923 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:57.190 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:12:57.190 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:12:57.190 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:12:57.190 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:12:57.190 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:12:57.190 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:12:57.518 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:57.519 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:12:57.900 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph 2024-03-25T16:12:58.382 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:12:58.408 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:12:58.409 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:58.409 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:12:58.409 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:12:58.409 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:12:58.409 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:58.472 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:10 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:12:58.624 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:12:58.626 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-25T16:12:58.626 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:12:58.627 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:12:58.628 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:12:58.629 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:12:58.630 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:58.630 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:12:58.630 INFO:teuthology.orchestra.run.smithi134.stdout:========================================================================================= 2024-03-25T16:12:58.630 INFO:teuthology.orchestra.run.smithi134.stdout:Install 36 Packages 2024-03-25T16:12:58.630 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:12:58.630 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 30 M 2024-03-25T16:12:58.630 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 107 M 2024-03-25T16:12:58.631 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:12:58.866 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:12:58.889 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:12:58.889 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:58.889 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:12:58.889 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:12:58.889 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:12:58.889 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:59.100 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:12:59.123 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:12:59.123 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:12:59.123 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:12:59.123 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:12:59.123 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:12:59.124 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:12:59.188 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:12:59.503 INFO:teuthology.orchestra.run.smithi134.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-25T16:13:00.328 INFO:teuthology.orchestra.run.smithi134.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-03-25T16:13:00.503 INFO:teuthology.orchestra.run.smithi134.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 2.1 MB 00:01 2024-03-25T16:13:00.629 INFO:teuthology.orchestra.run.smithi134.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.9 MB/s | 247 kB 00:00 2024-03-25T16:13:00.746 INFO:teuthology.orchestra.run.smithi134.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:13:00.779 INFO:teuthology.orchestra.run.smithi134.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T16:13:00.813 INFO:teuthology.orchestra.run.smithi134.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 935 kB/s | 32 kB 00:00 2024-03-25T16:13:00.847 INFO:teuthology.orchestra.run.smithi134.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:13:00.881 INFO:teuthology.orchestra.run.smithi134.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:13:00.922 INFO:teuthology.orchestra.run.smithi134.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:13:00.988 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:13:00.988 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:13:00.988 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:13:00.989 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:13:00.990 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:13:00.991 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:13:00.991 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:13:00.991 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:13:00.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:13:00.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:13:00.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:13:00.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:13:00.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:13:00.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:13:00.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:13:01.006 INFO:teuthology.orchestra.run.smithi134.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-25T16:13:01.048 INFO:teuthology.orchestra.run.smithi134.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:13:01.090 INFO:teuthology.orchestra.run.smithi134.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:13:01.131 INFO:teuthology.orchestra.run.smithi134.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:13:01.308 INFO:teuthology.orchestra.run.smithi134.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 707 kB/s | 125 kB 00:00 2024-03-25T16:13:01.382 INFO:teuthology.orchestra.run.smithi134.stdout:(16/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 2.5 MB/s | 4.7 MB 00:01 2024-03-25T16:13:01.541 INFO:teuthology.orchestra.run.smithi134.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 96 kB/s | 22 kB 00:00 2024-03-25T16:13:01.600 INFO:teuthology.orchestra.run.smithi134.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 696 kB/s | 151 kB 00:00 2024-03-25T16:13:01.683 INFO:teuthology.orchestra.run.smithi134.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T16:13:01.717 INFO:teuthology.orchestra.run.smithi134.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 248 kB/s | 43 kB 00:00 2024-03-25T16:13:01.767 INFO:teuthology.orchestra.run.smithi134.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.2 MB/s | 358 kB 00:00 2024-03-25T16:13:01.784 INFO:teuthology.orchestra.run.smithi134.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 159 kB/s | 11 kB 00:00 2024-03-25T16:13:01.809 INFO:teuthology.orchestra.run.smithi134.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-03-25T16:13:01.809 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:13:01.809 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:01.809 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:13:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:13:01.811 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:01.812 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:01.834 INFO:teuthology.orchestra.run.smithi134.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-03-25T16:13:01.851 INFO:teuthology.orchestra.run.smithi134.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 463 kB/s | 19 kB 00:00 2024-03-25T16:13:01.885 INFO:teuthology.orchestra.run.smithi134.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-03-25T16:13:01.902 INFO:teuthology.orchestra.run.smithi134.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-03-25T16:13:01.969 INFO:teuthology.orchestra.run.smithi134.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T16:13:01.994 INFO:teuthology.orchestra.run.smithi134.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 724 kB/s | 79 kB 00:00 2024-03-25T16:13:02.028 INFO:teuthology.orchestra.run.smithi134.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-25T16:13:02.070 INFO:teuthology.orchestra.run.smithi134.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T16:13:02.095 INFO:teuthology.orchestra.run.smithi134.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2024-03-25T16:13:02.162 INFO:teuthology.orchestra.run.smithi134.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-03-25T16:13:02.187 INFO:teuthology.orchestra.run.smithi134.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-25T16:13:02.195 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base 2024-03-25T16:13:02.212 INFO:teuthology.orchestra.run.smithi134.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-03-25T16:13:02.646 INFO:teuthology.orchestra.run.smithi134.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 7.2 MB/s | 17 MB 00:02 2024-03-25T16:13:02.649 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:02.649 INFO:teuthology.orchestra.run.smithi134.stdout:Total 7.4 MB/s | 30 MB 00:04 2024-03-25T16:13:02.674 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:02.714 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:02.714 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:02.765 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:15 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:02.862 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:02.913 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:02.914 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:02.914 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:02.987 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install cephadm 2024-03-25T16:13:03.113 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:03.114 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:03.551 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:16 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:03.700 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:03.701 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:03.701 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:13:03.701 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:03.701 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 747 k 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 754 k 2024-03-25T16:13:03.702 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:04.425 INFO:teuthology.orchestra.run.smithi159.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.0 MB/s | 747 kB 00:00 2024-03-25T16:13:04.426 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:04.426 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.0 MB/s | 747 kB 00:00 2024-03-25T16:13:04.426 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:04.432 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:04.432 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:04.438 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:04.438 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:04.460 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:04.511 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:04.523 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:13:04.562 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:13:04.592 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:13:04.663 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:13:04.699 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:13:04.732 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:13:04.824 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:13:04.901 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:13:04.943 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:13:04.993 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:05.023 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:13:05.077 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:13:05.080 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:05.115 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:13:05.172 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:13:05.228 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:13:05.413 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:13:05.461 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:13:05.499 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:13:05.501 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:05.534 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:13:05.564 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:13:05.632 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:13:05.666 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:13:05.699 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:13:05.739 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:13:05.774 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi159.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:05.808 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:05.817 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:13:05.906 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:13:05.940 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:13:06.012 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:13:06.028 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:13:06.149 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:13:06.284 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:13:06.458 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:13:06.486 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:13:06.511 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:06.602 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:19 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:06.755 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:13:06.756 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:06.757 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 145 k 2024-03-25T16:13:06.757 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 436 k 2024-03-25T16:13:06.757 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:07.114 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 407 kB/s | 145 kB 00:00 2024-03-25T16:13:07.115 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:07.115 INFO:teuthology.orchestra.run.smithi159.stdout:Total 405 kB/s | 145 kB 00:00 2024-03-25T16:13:07.115 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:07.123 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:07.123 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:07.179 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:07.179 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:07.315 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:07.382 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:07.406 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:07.406 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:07.407 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:13:07.407 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:07.996 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:13:08.023 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:13:08.023 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:08.023 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:13:08.023 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:13:08.023 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:13:08.023 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:08.389 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:08.389 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:08.390 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:08.390 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:08.390 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:08.390 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:08.463 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:13:08.487 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:13:08.487 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:08.487 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:13:08.487 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:13:08.488 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:13:08.488 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:08.644 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr 2024-03-25T16:13:08.731 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:13:08.754 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:13:08.754 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:08.754 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:13:08.754 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:13:08.755 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:13:08.755 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:08.813 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:13:09.218 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:22 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:09.316 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:09.368 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:09.369 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:09.443 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:13:10.004 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:23 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:10.152 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:13:10.154 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout:====================================================================================== 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout:Install 12 Packages 2024-03-25T16:13:10.155 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:10.156 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 6.9 M 2024-03-25T16:13:10.156 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 26 M 2024-03-25T16:13:10.156 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:10.422 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:13:10.423 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:13:10.424 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:13:10.425 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:13:10.535 INFO:teuthology.orchestra.run.smithi159.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 79 kB/s | 15 kB 00:00 2024-03-25T16:13:10.593 INFO:teuthology.orchestra.run.smithi159.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 124 kB/s | 31 kB 00:00 2024-03-25T16:13:10.702 INFO:teuthology.orchestra.run.smithi159.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 238 kB 00:00 2024-03-25T16:13:10.819 INFO:teuthology.orchestra.run.smithi159.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 7.2 MB/s | 863 kB 00:00 2024-03-25T16:13:10.923 INFO:teuthology.orchestra.run.smithi159.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.6 MB/s | 1.0 MB 00:00 2024-03-25T16:13:11.112 INFO:teuthology.orchestra.run.smithi159.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 101 kB/s | 19 kB 00:00 2024-03-25T16:13:11.187 INFO:teuthology.orchestra.run.smithi159.stdout:(7/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.1 MB/s | 1.8 MB 00:00 2024-03-25T16:13:11.271 INFO:teuthology.orchestra.run.smithi159.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T16:13:11.354 INFO:teuthology.orchestra.run.smithi159.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 368 kB/s | 31 kB 00:00 2024-03-25T16:13:11.429 INFO:teuthology.orchestra.run.smithi159.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-25T16:13:11.513 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:13:11.513 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:13:11.514 INFO:teuthology.orchestra.run.smithi134.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:13:11.515 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:13:11.516 INFO:teuthology.orchestra.run.smithi134.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:13:11.517 INFO:teuthology.orchestra.run.smithi134.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:13:11.517 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:13:11.517 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:13:11.517 INFO:teuthology.orchestra.run.smithi134.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:13:11.517 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:11.517 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:11.547 INFO:teuthology.orchestra.run.smithi159.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.7 MB/s | 2.0 MB 00:00 2024-03-25T16:13:11.571 INFO:teuthology.orchestra.run.smithi159.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 376 kB/s | 144 kB 00:00 2024-03-25T16:13:11.576 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:11.576 INFO:teuthology.orchestra.run.smithi159.stdout:Total 4.8 MB/s | 6.9 MB 00:01 2024-03-25T16:13:11.594 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:11.610 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:11.610 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:11.785 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:11.786 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:11.887 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base 2024-03-25T16:13:12.298 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:12.363 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:13:12.406 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:13:12.453 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:13:12.466 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:24 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:12.483 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:13:12.564 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:12.592 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:13:12.616 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:12.617 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:13:12.617 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:12.692 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install cephadm 2024-03-25T16:13:12.725 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:13:12.819 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:13:12.863 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:13:12.892 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:12.949 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:13:13.189 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:13:13.224 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:13.256 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:25 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:13.408 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:13.409 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:13.409 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:13:13.409 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:13.409 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:13.409 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:13:13.409 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:13.410 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:13.410 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:13.410 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:13:13.410 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:13.410 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 747 k 2024-03-25T16:13:13.410 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 754 k 2024-03-25T16:13:13.410 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:13:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:13:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:13:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:13:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:13:14.015 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:13:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:13:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:13:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:13:14.016 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:13:14.036 INFO:teuthology.orchestra.run.smithi134.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.2 MB/s | 747 kB 00:00 2024-03-25T16:13:14.036 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:14.037 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.2 MB/s | 747 kB 00:00 2024-03-25T16:13:14.037 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:14.042 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:14.042 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:14.048 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:14.049 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:14.120 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:14.348 INFO:teuthology.orchestra.run.smithi159.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:13:14.349 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:13:14.349 INFO:teuthology.orchestra.run.smithi159.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:13:14.349 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:13:14.350 INFO:teuthology.orchestra.run.smithi159.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:13:14.350 INFO:teuthology.orchestra.run.smithi159.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:13:14.350 INFO:teuthology.orchestra.run.smithi159.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:13:14.350 INFO:teuthology.orchestra.run.smithi159.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:13:14.350 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:14.350 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:14.568 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:14.592 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:13:14.657 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:15.040 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:15.156 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:28 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:15.306 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:15.307 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-25T16:13:15.307 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:13:15.307 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-25T16:13:15.307 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:15.307 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:13:15.307 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:15.308 INFO:teuthology.orchestra.run.smithi159.stdout:============================================================================================ 2024-03-25T16:13:15.309 INFO:teuthology.orchestra.run.smithi159.stdout:Install 12 Packages 2024-03-25T16:13:15.309 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:15.309 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 37 M 2024-03-25T16:13:15.309 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 209 M 2024-03-25T16:13:15.309 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:15.330 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:13:15.330 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:15.330 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:15.331 INFO:teuthology.orchestra.run.smithi134.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:15.331 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:15.331 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:15.557 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:13:15.611 INFO:teuthology.orchestra.run.smithi159.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 177 kB/s | 32 kB 00:00 2024-03-25T16:13:15.678 INFO:teuthology.orchestra.run.smithi159.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 261 kB/s | 17 kB 00:00 2024-03-25T16:13:15.737 INFO:teuthology.orchestra.run.smithi159.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 584 kB/s | 34 kB 00:00 2024-03-25T16:13:16.132 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:28 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:16.279 INFO:teuthology.orchestra.run.smithi159.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.5 MB/s | 3.0 MB 00:00 2024-03-25T16:13:16.286 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:16.287 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:13:16.288 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:16.288 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 145 k 2024-03-25T16:13:16.288 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 436 k 2024-03-25T16:13:16.288 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:16.749 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 315 kB/s | 145 kB 00:00 2024-03-25T16:13:16.750 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:16.750 INFO:teuthology.orchestra.run.smithi134.stdout:Total 314 kB/s | 145 kB 00:00 2024-03-25T16:13:16.750 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:16.758 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:16.758 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:16.811 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:16.811 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:16.960 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:17.017 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:17.042 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:17.042 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:17.042 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:13:17.043 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:17.062 INFO:teuthology.orchestra.run.smithi159.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 3.6 MB/s | 4.7 MB 00:01 2024-03-25T16:13:17.337 INFO:teuthology.orchestra.run.smithi159.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 794 kB/s | 217 kB 00:00 2024-03-25T16:13:17.421 INFO:teuthology.orchestra.run.smithi159.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.2 MB/s | 4.8 MB 00:01 2024-03-25T16:13:17.796 INFO:teuthology.orchestra.run.smithi159.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-25T16:13:17.897 INFO:teuthology.orchestra.run.smithi159.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.9 MB/s | 195 kB 00:00 2024-03-25T16:13:18.022 INFO:teuthology.orchestra.run.smithi159.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-03-25T16:13:18.056 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:13:18.056 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:18.056 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:18.056 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:18.056 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:18.056 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:18.305 INFO:teuthology.orchestra.run.smithi159.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-03-25T16:13:18.327 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr 2024-03-25T16:13:18.902 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:30 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:18.999 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:19.051 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:19.052 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:13:19.052 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:19.123 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:13:19.164 INFO:teuthology.orchestra.run.smithi159.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.0 MB/s | 7.4 MB 00:03 2024-03-25T16:13:19.167 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:19.167 INFO:teuthology.orchestra.run.smithi159.stdout:Total 9.6 MB/s | 37 MB 00:03 2024-03-25T16:13:19.212 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:19.237 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:19.237 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:19.596 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:19.596 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:19.681 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:31 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:19.829 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:19.830 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-25T16:13:19.830 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:13:19.831 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:13:19.832 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:13:19.832 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:13:19.832 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:19.832 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:19.832 INFO:teuthology.orchestra.run.smithi134.stdout:====================================================================================== 2024-03-25T16:13:19.832 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-03-25T16:13:19.832 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:19.833 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.9 M 2024-03-25T16:13:19.833 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-03-25T16:13:19.833 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:20.327 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:20.402 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 163 kB/s | 31 kB 00:00 2024-03-25T16:13:20.403 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:13:20.443 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:13:20.511 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 51 kB/s | 15 kB 00:00 2024-03-25T16:13:20.536 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:20.580 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:13:20.628 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T16:13:20.653 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T16:13:20.758 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:20.787 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:20.853 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.8 MB/s | 1.8 MB 00:00 2024-03-25T16:13:20.864 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:13:20.887 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.3 MB/s | 863 kB 00:00 2024-03-25T16:13:21.004 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 129 kB/s | 19 kB 00:00 2024-03-25T16:13:21.037 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.4 MB/s | 548 kB 00:00 2024-03-25T16:13:21.088 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 614 kB/s | 31 kB 00:00 2024-03-25T16:13:21.155 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:13:21.180 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 822 kB/s | 144 kB 00:00 2024-03-25T16:13:21.322 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.7 MB/s | 2.0 MB 00:00 2024-03-25T16:13:21.324 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:21.324 INFO:teuthology.orchestra.run.smithi134.stdout:Total 4.6 MB/s | 6.9 MB 00:01 2024-03-25T16:13:21.342 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:21.359 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:21.359 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:21.517 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:13:21.532 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:21.532 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:21.625 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:13:22.041 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:22.131 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:13:22.166 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:13:22.212 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:13:22.251 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:13:22.355 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:13:22.483 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:13:22.569 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:13:22.622 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:13:22.650 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:22.682 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:13:22.868 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:13:22.926 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:13:22.957 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:13:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:13:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:13:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:13:23.815 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:13:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:13:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:13:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:13:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:13:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:13:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:13:23.816 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:13:23.898 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:13:23.928 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:13:24.154 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:13:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:13:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:13:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:13:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:13:24.155 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:24.155 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:24.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:13:24.986 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:36 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:25.137 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:25.138 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-25T16:13:25.138 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:13:25.138 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-25T16:13:25.138 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:25.138 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:13:25.138 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:25.139 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:25.140 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================================ 2024-03-25T16:13:25.140 INFO:teuthology.orchestra.run.smithi134.stdout:Install 12 Packages 2024-03-25T16:13:25.140 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:25.140 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 37 M 2024-03-25T16:13:25.140 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 209 M 2024-03-25T16:13:25.140 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:25.744 INFO:teuthology.orchestra.run.smithi134.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 185 kB/s | 32 kB 00:00 2024-03-25T16:13:25.795 INFO:teuthology.orchestra.run.smithi134.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 345 kB/s | 17 kB 00:00 2024-03-25T16:13:25.853 INFO:teuthology.orchestra.run.smithi134.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 591 kB/s | 34 kB 00:00 2024-03-25T16:13:25.920 INFO:teuthology.orchestra.run.smithi134.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.6 MB/s | 3.0 MB 00:00 2024-03-25T16:13:26.212 INFO:teuthology.orchestra.run.smithi134.stdout:(5/12): ceph-mgr-diskprediction-local-19.0.0-14 12 MB/s | 7.4 MB 00:00 2024-03-25T16:13:26.287 INFO:teuthology.orchestra.run.smithi134.stdout:(6/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 11 MB/s | 4.7 MB 00:00 2024-03-25T16:13:26.354 INFO:teuthology.orchestra.run.smithi134.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-03-25T16:13:26.379 INFO:teuthology.orchestra.run.smithi134.stdout:(8/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.3 MB/s | 217 kB 00:00 2024-03-25T16:13:26.469 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:26.469 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:13:26.469 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:13:26.470 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:13:26.713 INFO:teuthology.orchestra.run.smithi134.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-25T16:13:26.813 INFO:teuthology.orchestra.run.smithi134.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 452 kB/s | 195 kB 00:00 2024-03-25T16:13:26.922 INFO:teuthology.orchestra.run.smithi134.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:13:26.926 INFO:teuthology.orchestra.run.smithi159.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:26.927 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:27.014 INFO:teuthology.orchestra.run.smithi134.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 803 kB 00:00 2024-03-25T16:13:27.015 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:27.016 INFO:teuthology.orchestra.run.smithi134.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-25T16:13:27.056 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:27.079 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:27.079 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:27.327 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:13:27.442 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:27.442 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:27.901 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:40 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:28.053 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:13:28.054 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout:Install 13 Packages 2024-03-25T16:13:28.055 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:28.056 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 2.0 M 2024-03-25T16:13:28.056 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 26 M 2024-03-25T16:13:28.056 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:28.148 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:28.229 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:13:28.237 INFO:teuthology.orchestra.run.smithi159.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 184 kB/s | 24 kB 00:00 2024-03-25T16:13:28.259 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:13:28.263 INFO:teuthology.orchestra.run.smithi159.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-25T16:13:28.313 INFO:teuthology.orchestra.run.smithi159.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:13:28.338 INFO:teuthology.orchestra.run.smithi159.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:13:28.353 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:28.355 INFO:teuthology.orchestra.run.smithi159.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 5.0 MB/s | 211 kB 00:00 2024-03-25T16:13:28.372 INFO:teuthology.orchestra.run.smithi159.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 43 kB 00:00 2024-03-25T16:13:28.396 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:13:28.449 INFO:teuthology.orchestra.run.smithi159.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 200 kB/s | 15 kB 00:00 2024-03-25T16:13:28.464 INFO:teuthology.orchestra.run.smithi159.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 296 kB/s | 32 kB 00:00 2024-03-25T16:13:28.556 INFO:teuthology.orchestra.run.smithi159.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.8 MB/s | 203 kB 00:00 2024-03-25T16:13:28.574 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:28.598 INFO:teuthology.orchestra.run.smithi159.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 106 kB/s | 52 kB 00:00 2024-03-25T16:13:28.623 INFO:teuthology.orchestra.run.smithi159.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-03-25T16:13:28.638 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:28.682 INFO:teuthology.orchestra.run.smithi159.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.7 MB/s | 1.0 MB 00:00 2024-03-25T16:13:28.722 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:13:28.732 INFO:teuthology.orchestra.run.smithi159.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 670 kB/s | 90 kB 00:00 2024-03-25T16:13:28.734 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:28.735 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.0 MB/s | 2.0 MB 00:00 2024-03-25T16:13:28.735 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:28.748 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:28.748 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:28.840 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:28.840 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:29.359 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:29.366 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:13:29.474 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:13:29.530 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:13:29.592 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:13:29.647 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:13:29.685 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:13:29.726 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:13:29.819 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:13:29.946 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:13:29.994 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:13:30.708 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:13:30.726 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:13:30.774 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:13:30.808 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:13:30.857 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:13:30.887 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:31.644 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:31.644 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:13:31.644 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:13:31.644 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:13:31.645 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:13:31.645 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:13:31.645 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:13:31.645 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:13:31.645 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:13:31.645 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:13:31.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:13:31.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:13:31.646 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:13:31.743 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:13:31.768 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:31.996 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:31.997 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:32.267 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:13:32.832 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:45 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:32.983 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:13:32.984 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout:Install 6 Packages 2024-03-25T16:13:32.985 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:32.986 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 6.8 M 2024-03-25T16:13:32.986 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 33 M 2024-03-25T16:13:32.986 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:33.642 INFO:teuthology.orchestra.run.smithi159.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-25T16:13:33.709 INFO:teuthology.orchestra.run.smithi159.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 620 kB/s | 248 kB 00:00 2024-03-25T16:13:33.759 INFO:teuthology.orchestra.run.smithi159.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 306 kB/s | 138 kB 00:00 2024-03-25T16:13:33.935 INFO:teuthology.orchestra.run.smithi159.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T16:13:33.960 INFO:teuthology.orchestra.run.smithi159.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 231 kB/s | 58 kB 00:00 2024-03-25T16:13:33.985 INFO:teuthology.orchestra.run.smithi159.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 226 kB/s | 51 kB 00:00 2024-03-25T16:13:33.989 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:33.989 INFO:teuthology.orchestra.run.smithi159.stdout:Total 6.8 MB/s | 6.8 MB 00:01 2024-03-25T16:13:34.002 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:34.013 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:34.013 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:34.103 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:34.104 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:34.291 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:13:34.291 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:13:34.291 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:13:34.292 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:13:34.417 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:34.576 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:13:34.620 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:13:34.755 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:34.756 INFO:teuthology.orchestra.run.smithi134.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:13:34.757 INFO:teuthology.orchestra.run.smithi134.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:13:34.757 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:34.757 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:35.112 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:13:35.152 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:13:35.258 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:13:35.326 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:13:35.349 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:13:35.691 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:47 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:35.842 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:13:35.843 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout:Install 13 Packages 2024-03-25T16:13:35.844 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:35.845 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 2.0 M 2024-03-25T16:13:35.845 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 26 M 2024-03-25T16:13:35.845 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:36.091 INFO:teuthology.orchestra.run.smithi134.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 172 kB/s | 24 kB 00:00 2024-03-25T16:13:36.117 INFO:teuthology.orchestra.run.smithi134.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 108 kB/s | 18 kB 00:00 2024-03-25T16:13:36.152 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:13:36.153 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:13:36.153 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:13:36.153 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:13:36.153 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:13:36.153 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:13:36.167 INFO:teuthology.orchestra.run.smithi134.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:13:36.192 INFO:teuthology.orchestra.run.smithi134.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:13:36.217 INFO:teuthology.orchestra.run.smithi134.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:13:36.243 INFO:teuthology.orchestra.run.smithi134.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-25T16:13:36.401 INFO:teuthology.orchestra.run.smithi134.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 116 kB/s | 52 kB 00:00 2024-03-25T16:13:36.460 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:13:36.460 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:36.460 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:36.460 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:36.460 INFO:teuthology.orchestra.run.smithi159.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:13:36.461 INFO:teuthology.orchestra.run.smithi159.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:13:36.461 INFO:teuthology.orchestra.run.smithi159.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:13:36.461 INFO:teuthology.orchestra.run.smithi159.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:13:36.461 INFO:teuthology.orchestra.run.smithi159.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:13:36.461 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:36.461 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:36.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse 2024-03-25T16:13:36.743 INFO:teuthology.orchestra.run.smithi134.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 61 kB/s | 32 kB 00:00 2024-03-25T16:13:36.768 INFO:teuthology.orchestra.run.smithi134.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 28 kB/s | 15 kB 00:00 2024-03-25T16:13:37.002 INFO:teuthology.orchestra.run.smithi134.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 253 kB/s | 59 kB 00:00 2024-03-25T16:13:37.185 INFO:teuthology.orchestra.run.smithi134.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 258 kB/s | 203 kB 00:00 2024-03-25T16:13:37.210 INFO:teuthology.orchestra.run.smithi134.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-03-25T16:13:37.276 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:50 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:37.428 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:37.429 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-03-25T16:13:37.429 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:37.429 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 964 k 2024-03-25T16:13:37.429 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.8 M 2024-03-25T16:13:37.429 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:37.570 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 557 kB/s | 79 kB 00:00 2024-03-25T16:13:38.219 INFO:teuthology.orchestra.run.smithi134.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 708 kB/s | 1.0 MB 00:01 2024-03-25T16:13:38.224 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:38.224 INFO:teuthology.orchestra.run.smithi134.stdout:Total 882 kB/s | 2.0 MB 00:02 2024-03-25T16:13:38.225 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:38.240 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:38.240 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:38.338 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:38.339 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:38.537 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 799 kB/s | 885 kB 00:01 2024-03-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi159.stdout:Total 868 kB/s | 964 kB 00:01 2024-03-25T16:13:38.538 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:38.560 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:38.635 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:38.635 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:38.810 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:38.845 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:38.955 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:13:39.016 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:13:39.029 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:39.070 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:13:39.126 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:13:39.163 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:13:39.204 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:13:39.298 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:13:39.406 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:13:39.463 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:13:39.757 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:39.757 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:13:40.080 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:13:40.081 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:40.081 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:40.081 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:13:40.081 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:40.081 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:40.188 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:13:40.244 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:13:40.278 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:13:40.343 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:13:40.353 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-volume 2024-03-25T16:13:40.372 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:40.937 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:53 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:41.088 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repository Size 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:41.089 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:41.090 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-03-25T16:13:41.090 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:41.090 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 332 k 2024-03-25T16:13:41.090 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.5 M 2024-03-25T16:13:41.090 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:41.217 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 544 kB/s | 69 kB 00:00 2024-03-25T16:13:41.218 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:13:41.218 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:13:41.219 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:13:41.484 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 670 kB/s | 264 kB 00:00 2024-03-25T16:13:41.485 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:41.485 INFO:teuthology.orchestra.run.smithi159.stdout:Total 840 kB/s | 332 kB 00:00 2024-03-25T16:13:41.485 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:41.494 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:41.494 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:41.530 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:41.531 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:41.605 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:13:41.606 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:13:41.607 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:13:41.607 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:13:41.607 INFO:teuthology.orchestra.run.smithi134.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:13:41.607 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:41.607 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:41.680 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:41.858 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:13:41.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:13:41.903 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:13:41.919 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:13:41.920 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:41.920 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:13:41.920 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:42.449 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:54 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:42.601 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:13:42.602 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:13:42.603 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:42.603 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:42.603 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:13:42.603 INFO:teuthology.orchestra.run.smithi134.stdout:Install 6 Packages 2024-03-25T16:13:42.603 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:42.603 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 6.8 M 2024-03-25T16:13:42.603 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 33 M 2024-03-25T16:13:42.604 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:42.679 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:13:42.992 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:13:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:42.993 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:13:42.993 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:42.993 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:43.137 INFO:teuthology.orchestra.run.smithi134.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-25T16:13:43.162 INFO:teuthology.orchestra.run.smithi134.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 550 kB/s | 248 kB 00:00 2024-03-25T16:13:43.323 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel 2024-03-25T16:13:43.329 INFO:teuthology.orchestra.run.smithi134.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 223 kB/s | 138 kB 00:00 2024-03-25T16:13:43.480 INFO:teuthology.orchestra.run.smithi134.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 182 kB/s | 58 kB 00:00 2024-03-25T16:13:43.555 INFO:teuthology.orchestra.run.smithi134.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 226 kB/s | 51 kB 00:00 2024-03-25T16:13:43.655 INFO:teuthology.orchestra.run.smithi134.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T16:13:43.659 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:43.659 INFO:teuthology.orchestra.run.smithi134.stdout:Total 6.4 MB/s | 6.8 MB 00:01 2024-03-25T16:13:43.672 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:43.683 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:43.684 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:43.772 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:43.773 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:43.906 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:56 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:44.056 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:44.057 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:44.058 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:13:44.058 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:44.058 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 127 k 2024-03-25T16:13:44.058 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 456 k 2024-03-25T16:13:44.058 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:44.074 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:44.245 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:13:44.298 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:13:44.465 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 311 kB/s | 127 kB 00:00 2024-03-25T16:13:44.465 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:44.466 INFO:teuthology.orchestra.run.smithi159.stdout:Total 310 kB/s | 127 kB 00:00 2024-03-25T16:13:44.466 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:44.471 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:44.471 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:44.528 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:44.529 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:44.668 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:44.784 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:44.812 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:13:44.892 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:13:44.953 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:13:44.977 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:13:45.234 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:45.502 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:45.502 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:45.503 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:45.503 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:45.503 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:45.503 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:45.771 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2 2024-03-25T16:13:45.828 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:13:45.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:13:45.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:13:45.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:13:45.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:13:45.828 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:13:46.134 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:13:46.135 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:46.136 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:46.353 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:01:59 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:46.399 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse 2024-03-25T16:13:46.455 INFO:teuthology.orchestra.run.smithi159.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:46.509 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:46.510 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:46.510 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:46.577 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs-devel 2024-03-25T16:13:46.966 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:01:58 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:47.121 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-25T16:13:47.122 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:47.123 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 964 k 2024-03-25T16:13:47.123 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.8 M 2024-03-25T16:13:47.123 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:47.146 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:00 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:47.298 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:47.299 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:13:47.299 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:47.299 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:47.299 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 31 k 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 140 k 2024-03-25T16:13:47.300 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:47.334 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 376 kB/s | 79 kB 00:00 2024-03-25T16:13:47.603 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 102 kB/s | 31 kB 00:00 2024-03-25T16:13:47.604 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:47.604 INFO:teuthology.orchestra.run.smithi159.stdout:Total 102 kB/s | 31 kB 00:00 2024-03-25T16:13:47.604 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:47.608 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:47.608 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:47.635 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:47.635 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:47.770 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:47.861 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:48.016 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 992 kB/s | 885 kB 00:00 2024-03-25T16:13:48.017 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:48.017 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.1 MB/s | 964 kB 00:00 2024-03-25T16:13:48.018 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:48.038 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:48.039 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:48.115 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:48.115 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:48.295 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:48.296 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:48.443 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:13:48.516 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:48.549 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:48.549 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:48.550 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:48.702 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2 2024-03-25T16:13:49.271 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:13:49.271 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:13:49.276 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:49.374 INFO:teuthology.orchestra.run.smithi159.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:49.425 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:49.426 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:49.426 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:49.475 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1 2024-03-25T16:13:49.568 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:13:49.568 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:49.569 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:49.569 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:13:49.569 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:49.569 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:49.874 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-volume 2024-03-25T16:13:50.044 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:03 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:50.141 INFO:teuthology.orchestra.run.smithi159.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:50.194 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:50.195 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:50.195 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:50.242 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados 2024-03-25T16:13:50.455 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:02 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:50.605 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:50.606 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:50.607 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repository Size 2024-03-25T16:13:50.607 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:50.607 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:50.607 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:13:50.607 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:13:50.607 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:13:50.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:50.608 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:50.608 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:50.608 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-25T16:13:50.608 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:50.608 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 332 k 2024-03-25T16:13:50.609 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.5 M 2024-03-25T16:13:50.609 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:50.770 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 423 kB/s | 69 kB 00:00 2024-03-25T16:13:50.810 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:03 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:50.906 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:50.958 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:50.958 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:50.959 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:51.008 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw 2024-03-25T16:13:51.537 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 284 kB/s | 264 kB 00:00 2024-03-25T16:13:51.538 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:51.539 INFO:teuthology.orchestra.run.smithi134.stdout:Total 356 kB/s | 332 kB 00:00 2024-03-25T16:13:51.539 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:51.548 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:51.548 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:51.567 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:04 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:51.585 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:51.585 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:51.662 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:51.714 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:51.715 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:51.715 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:51.732 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:51.756 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs 2024-03-25T16:13:51.919 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:13:51.964 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:13:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:13:52.261 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:52.261 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:13:52.261 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:52.322 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:52.419 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:52.470 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:52.470 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:52.471 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:52.516 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd 2024-03-25T16:13:52.673 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:13:53.026 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:13:53.026 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:53.026 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:53.026 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:13:53.026 INFO:teuthology.orchestra.run.smithi134.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:13:53.026 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:53.026 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:53.078 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:06 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:53.174 INFO:teuthology.orchestra.run.smithi159.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:53.225 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:53.226 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:13:53.226 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:53.272 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse 2024-03-25T16:13:53.419 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel 2024-03-25T16:13:53.850 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:06 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:53.999 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:05 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:54.007 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:13:54.008 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:54.015 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 85 k 2024-03-25T16:13:54.015 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 231 k 2024-03-25T16:13:54.015 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:54.151 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:54.152 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:54.152 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:13:54.152 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:54.152 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:54.152 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:13:54.152 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:54.153 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:54.153 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:54.153 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:13:54.153 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:54.153 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 127 k 2024-03-25T16:13:54.153 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 456 k 2024-03-25T16:13:54.153 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:54.665 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 130 kB/s | 85 kB 00:00 2024-03-25T16:13:54.666 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:54.667 INFO:teuthology.orchestra.run.smithi159.stdout:Total 130 kB/s | 85 kB 00:00 2024-03-25T16:13:54.667 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:54.679 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:54.679 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:54.742 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:54.746 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:54.850 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 182 kB/s | 127 kB 00:00 2024-03-25T16:13:54.850 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:54.851 INFO:teuthology.orchestra.run.smithi134.stdout:Total 181 kB/s | 127 kB 00:00 2024-03-25T16:13:54.851 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:54.858 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:54.858 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:54.884 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:54.915 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:54.915 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:55.014 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:55.061 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:55.159 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:55.457 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:55.655 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:55.732 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:13:55.732 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:55.733 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:55.733 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:13:55.933 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:55.933 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:55.933 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:55.933 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:55.933 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:55.933 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:55.934 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror 2024-03-25T16:13:56.155 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2 2024-03-25T16:13:56.508 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:09 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:13:56.663 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:13:56.663 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:56.663 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 3.1 M 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 13 M 2024-03-25T16:13:56.664 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:13:56.736 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:08 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:56.836 INFO:teuthology.orchestra.run.smithi134.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:56.890 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:56.891 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:13:56.892 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:56.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs-devel 2024-03-25T16:13:57.553 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:09 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:57.705 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 31 k 2024-03-25T16:13:57.706 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 140 k 2024-03-25T16:13:57.707 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:13:58.185 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 65 kB/s | 31 kB 00:00 2024-03-25T16:13:58.186 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:58.186 INFO:teuthology.orchestra.run.smithi134.stdout:Total 65 kB/s | 31 kB 00:00 2024-03-25T16:13:58.186 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:13:58.190 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:13:58.190 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:13:58.223 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:13:58.223 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:13:58.365 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:13:58.484 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:58.854 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:58.930 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.4 MB/s | 3.1 MB 00:02 2024-03-25T16:13:58.931 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:13:58.931 INFO:teuthology.orchestra.run.smithi159.stdout:Total 1.4 MB/s | 3.1 MB 00:02 2024-03-25T16:13:58.931 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:13:58.939 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:13:58.939 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:13:59.017 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:13:59.017 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:13:59.115 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:59.115 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:59.115 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:13:59.115 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:13:59.116 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:13:59.116 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:13:59.270 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2 2024-03-25T16:13:59.447 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:13:59.510 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:59.535 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:13:59.535 INFO:teuthology.orchestra.run.smithi159.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:13:59.535 INFO:teuthology.orchestra.run.smithi159.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:13:59.536 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:13:59.536 INFO:teuthology.orchestra.run.smithi159.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:13:59.536 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:13:59.840 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:11 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:13:59.938 INFO:teuthology.orchestra.run.smithi134.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:13:59.990 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:13:59.991 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:13:59.991 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:00.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1 2024-03-25T16:14:00.620 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:12 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:00.719 INFO:teuthology.orchestra.run.smithi134.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:00.742 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:00.742 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:00.742 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:14:00.742 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:00.742 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:00.742 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:14:00.772 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:00.773 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:14:00.773 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:00.821 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados 2024-03-25T16:14:01.001 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd 2024-03-25T16:14:01.388 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:13 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:01.485 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:01.536 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:01.537 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:14:01.537 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:01.586 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:14:01.594 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw 2024-03-25T16:14:01.734 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:14:01.735 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:01.735 INFO:teuthology.orchestra.run.smithi159.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:01.735 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 171 k 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 494 k 2024-03-25T16:14:01.736 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:14:02.161 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:02.258 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:02.311 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:02.312 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:14:02.312 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:02.364 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs 2024-03-25T16:14:02.388 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 263 kB/s | 171 kB 00:00 2024-03-25T16:14:02.389 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:02.389 INFO:teuthology.orchestra.run.smithi159.stdout:Total 262 kB/s | 171 kB 00:00 2024-03-25T16:14:02.389 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:14:02.397 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:14:02.397 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:14:02.455 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:14:02.455 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:14:02.618 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:14:02.730 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:02.932 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:14 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:03.030 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:03.080 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:03.081 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:14:03.082 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:03.134 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd 2024-03-25T16:14:03.159 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:03.462 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:03.462 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:03.462 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:14:03.462 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:03.462 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:03.462 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:14:03.685 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:14:03.700 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:15 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:03.798 INFO:teuthology.orchestra.run.smithi134.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:14:03.849 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:03.850 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:14:03.850 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:03.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse 2024-03-25T16:14:04.268 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:02:17 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:14:04.421 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout:Installing dependencies: 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout:Install 2 Packages 2024-03-25T16:14:04.422 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:04.423 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 91 M 2024-03-25T16:14:04.423 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 498 M 2024-03-25T16:14:04.423 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:14:04.473 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:16 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:04.621 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:04.622 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 85 k 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 231 k 2024-03-25T16:14:04.623 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:14:05.214 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 144 kB/s | 85 kB 00:00 2024-03-25T16:14:05.214 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:05.215 INFO:teuthology.orchestra.run.smithi134.stdout:Total 144 kB/s | 85 kB 00:00 2024-03-25T16:14:05.215 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:14:05.222 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:14:05.222 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:14:05.277 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:14:05.278 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:14:05.416 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:14:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:05.875 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:06.163 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:06.163 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:06.163 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:14:06.163 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:06.163 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:06.163 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:06.406 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror 2024-03-25T16:14:06.985 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:18 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:07.139 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:07.140 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 3.1 M 2024-03-25T16:14:07.141 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 13 M 2024-03-25T16:14:07.141 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:14:08.832 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 1.8 MB/s | 3.1 MB 00:01 2024-03-25T16:14:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.8 MB/s | 3.1 MB 00:01 2024-03-25T16:14:08.833 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:14:08.844 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:14:08.844 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:14:08.923 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:14:08.923 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:14:09.359 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:14:09.424 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:09.448 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:09.448 INFO:teuthology.orchestra.run.smithi134.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:14:09.448 INFO:teuthology.orchestra.run.smithi134.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:14:09.448 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:14:09.449 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:14:09.449 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:10.539 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:10.539 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:10.539 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:14:10.539 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:10.539 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:10.539 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:10.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd 2024-03-25T16:14:11.371 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:23 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:11.528 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout: Package Architecture Version Repository Size 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 171 k 2024-03-25T16:14:11.529 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 494 k 2024-03-25T16:14:11.530 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:14:12.176 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 265 kB/s | 171 kB 00:00 2024-03-25T16:14:12.177 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:12.177 INFO:teuthology.orchestra.run.smithi134.stdout:Total 265 kB/s | 171 kB 00:00 2024-03-25T16:14:12.177 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:14:12.186 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:14:12.186 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:14:12.244 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:14:12.244 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:14:12.395 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:14:12.511 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:12.957 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:13.245 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:14:13.245 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:13.245 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:14:13.246 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:13.246 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:13.246 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:14:13.462 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:14:14.039 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:02:26 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:14:14.196 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:14:14.196 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout:Installing dependencies: 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout:Install 2 Packages 2024-03-25T16:14:14.197 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:14:14.198 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 91 M 2024-03-25T16:14:14.198 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 498 M 2024-03-25T16:14:14.198 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:14:14.224 INFO:teuthology.orchestra.run.smithi159.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 2.3 MB/s | 22 MB 00:09 2024-03-25T16:14:30.277 INFO:teuthology.orchestra.run.smithi134.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 1.4 MB/s | 22 MB 00:16 2024-03-25T16:14:32.588 INFO:teuthology.orchestra.run.smithi159.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 2.4 MB/s | 68 MB 00:28 2024-03-25T16:14:32.590 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:32.590 INFO:teuthology.orchestra.run.smithi159.stdout:Total 3.2 MB/s | 91 MB 00:28 2024-03-25T16:14:32.591 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:14:32.596 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:14:32.596 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:14:33.193 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:14:33.194 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:14:36.308 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:14:43.408 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:14:43.497 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:14:46.997 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:14:46.999 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:14:47.265 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:14:47.266 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:47.266 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:14:47.266 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:47.266 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:14:47.266 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:47.266 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:14:47.495 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:14:47.975 INFO:teuthology.orchestra.run.smithi134.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 2.0 MB/s | 68 MB 00:33 2024-03-25T16:14:47.977 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:14:47.977 INFO:teuthology.orchestra.run.smithi134.stdout:Total 2.7 MB/s | 91 MB 00:33 2024-03-25T16:14:47.978 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:14:47.982 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:14:47.982 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:14:48.069 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:03:01 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:14:48.229 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:14:48.229 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:48.229 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 416 M 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.7 G 2024-03-25T16:14:48.230 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:14:48.564 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:14:48.564 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:14:51.642 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:14:58.720 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:14:58.799 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:15:02.110 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:15:02.369 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:15:02.369 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:15:02.369 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:15:02.369 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:02.369 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:15:02.369 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:15:02.370 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:15:02.615 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:15:03.179 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:03:15 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:15:03.328 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:15:03.329 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:15:03.329 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:15:03.329 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 416 M 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.7 G 2024-03-25T16:15:03.330 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:17:21.891 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 2.7 MB/s | 416 MB 02:33 2024-03-25T16:17:21.892 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:21.892 INFO:teuthology.orchestra.run.smithi159.stdout:Total 2.7 MB/s | 416 MB 02:33 2024-03-25T16:17:21.892 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:17:21.896 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:17:21.896 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:17:24.384 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:17:24.384 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:18:06.826 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:18:06.934 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:11.597 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:11.852 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:18:11.852 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:11.852 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:18:11.852 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:11.852 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:11.852 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:18:12.101 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:18:12.670 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:06:25 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:18:12.819 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:18:12.820 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:18:12.820 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:18:12.820 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:18:12.820 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout:======================================================================================== 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.4 M 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 6.4 M 2024-03-25T16:18:12.821 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:18:14.314 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 979 kB/s | 1.4 MB 00:01 2024-03-25T16:18:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:18:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Total 978 kB/s | 1.4 MB 00:01 2024-03-25T16:18:14.315 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:18:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:18:14.319 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:18:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:18:14.362 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:18:14.618 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:18:14.713 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:18:15.142 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:18:15.435 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:18:15.435 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:15.435 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:18:15.435 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:18:15.436 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:15.436 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:18:15.607 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:18:16.174 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:06:29 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:18:16.323 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:18:16.323 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 176 M 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 753 M 2024-03-25T16:18:16.324 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:21:47.149 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 853 kB/s | 176 MB 03:30 2024-03-25T16:21:47.150 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:21:47.150 INFO:teuthology.orchestra.run.smithi159.stdout:Total 853 kB/s | 176 MB 03:30 2024-03-25T16:21:47.150 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:21:47.153 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:21:47.153 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:21:48.216 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:21:48.216 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:22:04.705 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 1.0 MB/s | 416 MB 07:01 2024-03-25T16:22:04.706 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:22:04.706 INFO:teuthology.orchestra.run.smithi134.stdout:Total 1.0 MB/s | 416 MB 07:01 2024-03-25T16:22:04.706 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:22:04.710 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:22:04.710 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:22:06.137 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:22:06.227 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:22:07.173 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:22:07.174 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:22:10.961 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:22:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:22:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:22:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:22:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:11.230 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:22:11.230 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:22:11.420 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:22:11.987 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:10:24 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:22:12.137 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:22:12.137 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 672 M 2024-03-25T16:22:12.138 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 3.1 G 2024-03-25T16:22:12.139 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:22:49.424 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:22:49.532 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:22:56.458 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:22:56.721 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:22:56.721 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:22:56.721 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:22:56.721 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:22:56.721 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:22:56.721 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:22:56.972 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:22:57.545 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:11:09 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:22:57.694 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:22:57.694 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:22:57.694 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:22:57.694 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout:======================================================================================== 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.4 M 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 6.4 M 2024-03-25T16:22:57.695 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:23:01.477 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 386 kB/s | 1.4 MB 00:03 2024-03-25T16:23:01.478 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:01.478 INFO:teuthology.orchestra.run.smithi134.stdout:Total 386 kB/s | 1.4 MB 00:03 2024-03-25T16:23:01.479 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:23:01.482 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:23:01.483 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:23:01.530 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:23:01.531 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:23:01.784 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:23:01.876 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:23:02.318 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:23:02.590 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:23:02.590 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:23:02.590 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:23:02.590 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:02.590 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:23:02.590 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:23:02.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:23:03.359 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:11:15 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:23:03.509 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:23:03.509 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 176 M 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 753 M 2024-03-25T16:23:03.510 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:27:31.027 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 672 kB/s | 176 MB 04:27 2024-03-25T16:27:31.028 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:31.028 INFO:teuthology.orchestra.run.smithi134.stdout:Total 672 kB/s | 176 MB 04:27 2024-03-25T16:27:31.028 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:27:31.031 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:27:31.031 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:27:32.087 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:27:32.087 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:27:50.338 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:27:50.522 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:27:55.030 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:27:55.284 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:27:55.284 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:27:55.284 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:27:55.284 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:55.285 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:27:55.285 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:27:55.510 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:27:56.077 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:16:08 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:27:56.228 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:27:56.228 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 672 M 2024-03-25T16:27:56.229 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 3.1 G 2024-03-25T16:27:56.230 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:53:09.927 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 371 kB/s | 672 MB 30:57 2024-03-25T16:53:09.946 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:53:09.946 INFO:teuthology.orchestra.run.smithi159.stdout:Total 371 kB/s | 672 MB 30:57 2024-03-25T16:53:09.946 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:53:09.947 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:53:09.947 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:53:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:53:13.894 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:53:23.287 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 451 kB/s | 672 MB 25:27 2024-03-25T16:53:23.288 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:53:23.288 INFO:teuthology.orchestra.run.smithi134.stdout:Total 451 kB/s | 672 MB 25:27 2024-03-25T16:53:23.288 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:53:23.293 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:53:23.293 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:53:27.244 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:53:27.244 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:54:24.680 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:54:24.861 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:54:32.484 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:54:32.766 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:54:32.766 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:54:32.767 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:54:32.767 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:54:32.767 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:54:32.767 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:54:33.032 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:54:33.597 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:42:46 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:54:33.693 INFO:teuthology.orchestra.run.smithi159.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:54:33.745 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:54:33.746 INFO:teuthology.orchestra.run.smithi159.stdout:Nothing to do. 2024-03-25T16:54:33.746 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:54:33.808 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T16:54:34.376 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:42:47 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:54:34.524 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:54:34.525 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 20 M 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 103 M 2024-03-25T16:54:34.526 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:54:37.888 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:54:38.270 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:54:47.262 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:54:47.521 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T16:54:47.521 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:54:47.521 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:54:47.521 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:54:47.521 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:54:47.522 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:54:47.806 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:54:48.379 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:43:00 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:54:48.481 INFO:teuthology.orchestra.run.smithi134.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:54:48.536 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:54:48.537 INFO:teuthology.orchestra.run.smithi134.stdout:Nothing to do. 2024-03-25T16:54:48.537 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:54:48.616 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T16:54:49.205 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:43:01 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:54:49.358 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:54:49.359 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:54:49.359 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:54:49.359 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:54:49.359 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:54:49.359 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 20 M 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 103 M 2024-03-25T16:54:49.360 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:55:25.141 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 399 kB/s | 20 MB 00:50 2024-03-25T16:55:25.142 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:55:25.142 INFO:teuthology.orchestra.run.smithi159.stdout:Total 399 kB/s | 20 MB 00:50 2024-03-25T16:55:25.143 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:55:25.146 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:55:25.147 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:55:25.297 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:55:25.297 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:55:27.522 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:55:27.615 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:55:28.540 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:55:28.807 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:55:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:55:28.808 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:55:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:55:28.808 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:55:28.808 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:55:28.975 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T16:55:29.544 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:43:42 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:55:29.692 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:55:29.693 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:55:29.694 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:55:29.694 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 48 M 2024-03-25T16:55:29.694 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 173 M 2024-03-25T16:55:29.694 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:55:50.074 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 332 kB/s | 20 MB 01:00 2024-03-25T16:55:50.075 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:55:50.075 INFO:teuthology.orchestra.run.smithi134.stdout:Total 332 kB/s | 20 MB 01:00 2024-03-25T16:55:50.075 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:55:50.079 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:55:50.079 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:55:50.229 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:55:50.229 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:55:52.458 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:55:52.656 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:55:53.590 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:55:53.834 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:55:53.834 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:55:53.834 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:55:53.834 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:55:53.834 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:55:53.834 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:55:54.023 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T16:55:54.591 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:44:06 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:55:54.739 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:55:54.739 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:55:54.740 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:55:54.741 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 48 M 2024-03-25T16:55:54.741 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 173 M 2024-03-25T16:55:54.741 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T16:57:49.435 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 350 kB/s | 48 MB 02:19 2024-03-25T16:57:49.435 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:49.436 INFO:teuthology.orchestra.run.smithi159.stdout:Total 350 kB/s | 48 MB 02:19 2024-03-25T16:57:49.436 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T16:57:49.439 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T16:57:49.439 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T16:57:49.751 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T16:57:49.751 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T16:57:54.479 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T16:57:54.640 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:57:56.020 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:57:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:57:56.285 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:57:56.286 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T16:57:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:57:56.286 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:57:56.286 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T16:57:56.465 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T16:57:57.033 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:46:10 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T16:57:57.182 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 99 M 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 449 M 2024-03-25T16:57:57.183 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T16:57:58.834 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 394 kB/s | 48 MB 02:04 2024-03-25T16:57:58.835 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:57:58.835 INFO:teuthology.orchestra.run.smithi134.stdout:Total 394 kB/s | 48 MB 02:04 2024-03-25T16:57:58.835 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T16:57:58.839 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T16:57:58.839 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T16:57:59.150 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T16:57:59.150 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T16:58:03.871 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T16:58:03.964 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:58:05.273 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:58:05.536 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T16:58:05.536 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:58:05.536 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T16:58:05.536 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:05.536 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:58:05.536 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T16:58:05.721 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T16:58:06.283 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:46:18 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T16:58:06.432 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T16:58:06.433 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:58:06.433 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T16:58:06.433 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:58:06.433 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T16:58:06.433 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T16:58:06.433 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:58:06.434 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T16:58:06.434 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T16:58:06.434 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T16:58:06.434 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T16:58:06.434 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 99 M 2024-03-25T16:58:06.434 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 449 M 2024-03-25T16:58:06.435 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:03:35.201 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 298 kB/s | 99 MB 05:38 2024-03-25T17:03:35.202 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:03:35.202 INFO:teuthology.orchestra.run.smithi159.stdout:Total 298 kB/s | 99 MB 05:38 2024-03-25T17:03:35.202 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:03:35.205 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:03:35.205 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:03:35.817 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:03:35.817 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:03:46.079 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:03:46.173 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:03:49.069 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:03:49.348 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:03:49.348 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:03:49.348 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:03:49.348 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:03:49.348 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:03:49.348 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:03:49.506 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:03:50.075 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 0:52:03 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:03:50.222 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:03:50.223 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:03:50.224 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:03:50.224 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:03:50.224 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 287 M 2024-03-25T17:03:50.224 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.2 G 2024-03-25T17:03:50.224 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:04:02.613 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 283 kB/s | 99 MB 05:56 2024-03-25T17:04:02.613 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:04:02.614 INFO:teuthology.orchestra.run.smithi134.stdout:Total 283 kB/s | 99 MB 05:56 2024-03-25T17:04:02.614 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:04:02.617 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:04:02.618 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:04:03.228 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:04:03.228 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:04:13.457 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:04:13.561 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:04:16.425 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:04:16.688 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:04:16.688 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:04:16.689 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:04:16.689 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:16.689 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:04:16.689 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:04:16.883 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:04:17.450 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 0:52:29 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:04:17.598 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:04:17.599 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:04:17.599 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:04:17.599 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:04:17.599 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:04:17.599 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:04:17.599 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:04:17.600 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:04:17.600 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:04:17.600 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:04:17.600 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:04:17.600 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 287 M 2024-03-25T17:04:17.600 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.2 G 2024-03-25T17:04:17.600 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:15:01.266 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 456 kB/s | 287 MB 10:43 2024-03-25T17:15:01.267 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:01.267 INFO:teuthology.orchestra.run.smithi134.stdout:Total 456 kB/s | 287 MB 10:43 2024-03-25T17:15:01.267 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:15:01.269 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:15:01.270 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:15:02.973 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:15:02.973 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:15:32.365 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:15:32.460 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:15:33.602 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:15:33.906 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:15:33.906 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:33.906 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:15:33.906 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:33.906 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:33.906 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:15:34.124 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:15:34.691 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:03:46 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:15:34.843 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:15:34.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:15:34.844 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:15:34.844 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:15:34.844 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.9 M 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 28 M 2024-03-25T17:15:34.845 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:15:56.262 INFO:teuthology.orchestra.run.smithi134.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 284 kB/s | 5.9 MB 00:21 2024-03-25T17:15:56.263 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:56.263 INFO:teuthology.orchestra.run.smithi134.stdout:Total 284 kB/s | 5.9 MB 00:21 2024-03-25T17:15:56.263 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:15:56.267 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:15:56.267 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:15:56.339 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:15:56.339 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:15:57.081 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:15:57.187 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:15:57.765 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:15:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:15:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:58.036 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:15:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:58.036 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:58.037 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:15:58.236 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:15:58.799 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:04:10 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:15:58.951 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:15:58.951 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:15:58.951 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 109 k 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 440 k 2024-03-25T17:15:58.952 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:16:00.057 INFO:teuthology.orchestra.run.smithi134.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 98 kB/s | 109 kB 00:01 2024-03-25T17:16:00.057 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:16:00.057 INFO:teuthology.orchestra.run.smithi134.stdout:Total 98 kB/s | 109 kB 00:01 2024-03-25T17:16:00.058 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:16:00.061 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:16:00.061 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:16:00.097 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:16:00.097 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:16:00.239 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:16:00.346 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:16:00.731 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:16:01.000 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:16:01.001 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:01.001 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:16:01.001 INFO:teuthology.orchestra.run.smithi134.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:01.001 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:01.001 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:16:01.203 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:16:01.769 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:04:13 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:16:01.918 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:01.920 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 5.4 M 2024-03-25T17:16:01.921 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 25 M 2024-03-25T17:16:01.921 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:16:21.615 INFO:teuthology.orchestra.run.smithi134.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 278 kB/s | 5.4 MB 00:19 2024-03-25T17:16:21.616 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:16:21.616 INFO:teuthology.orchestra.run.smithi134.stdout:Total 278 kB/s | 5.4 MB 00:19 2024-03-25T17:16:21.616 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:16:21.620 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:16:21.620 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:16:21.688 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:16:21.688 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:16:22.350 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:16:22.438 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:16:22.923 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:16:23.189 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:16:23.189 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:23.189 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:16:23.189 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:23.189 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:23.189 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:16:23.424 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librados2-debuginfo 2024-03-25T17:16:23.986 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:04:35 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:16:24.134 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:16:24.135 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:16:24.136 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:16:24.136 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 44 M 2024-03-25T17:16:24.136 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 253 M 2024-03-25T17:16:24.136 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:18:38.066 INFO:teuthology.orchestra.run.smithi134.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 339 kB/s | 44 MB 02:13 2024-03-25T17:18:38.088 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:18:38.088 INFO:teuthology.orchestra.run.smithi134.stdout:Total 339 kB/s | 44 MB 02:13 2024-03-25T17:18:38.088 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:18:38.088 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:18:38.088 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:18:38.369 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:18:38.370 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:18:43.391 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:18:43.547 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:18:45.295 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:18:45.566 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:18:45.566 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:18:45.566 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:18:45.566 INFO:teuthology.orchestra.run.smithi134.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:18:45.566 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:18:45.566 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:18:45.771 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:18:46.336 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:06:58 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:18:46.485 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:18:46.486 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:18:46.486 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 52 M 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 308 M 2024-03-25T17:18:46.487 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:19:24.120 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 314 kB/s | 287 MB 15:33 2024-03-25T17:19:24.121 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:19:24.121 INFO:teuthology.orchestra.run.smithi159.stdout:Total 314 kB/s | 287 MB 15:33 2024-03-25T17:19:24.122 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:19:24.125 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:19:24.125 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:19:25.843 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:19:25.843 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:19:55.456 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:19:55.558 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:19:56.699 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:19:56.970 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:19:56.971 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:19:56.971 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:19:56.971 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:19:56.971 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:19:56.971 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:19:57.164 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:19:57.752 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:08:10 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:19:57.910 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:19:57.911 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:19:57.911 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:19:57.911 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:19:57.911 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 5.9 M 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 28 M 2024-03-25T17:19:57.912 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:20:11.470 INFO:teuthology.orchestra.run.smithi159.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 449 kB/s | 5.9 MB 00:13 2024-03-25T17:20:11.470 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:20:11.471 INFO:teuthology.orchestra.run.smithi159.stdout:Total 449 kB/s | 5.9 MB 00:13 2024-03-25T17:20:11.471 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:20:11.476 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:20:11.476 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:20:11.548 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:20:11.548 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:20:12.276 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:20:12.409 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:20:13.012 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:20:13.297 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:20:13.297 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:13.297 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:20:13.297 INFO:teuthology.orchestra.run.smithi159.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:13.297 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:13.297 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:20:13.475 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:20:14.045 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:08:27 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:20:14.197 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:20:14.197 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:14.197 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 109 k 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 440 k 2024-03-25T17:20:14.198 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:20:15.156 INFO:teuthology.orchestra.run.smithi159.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 114 kB/s | 109 kB 00:00 2024-03-25T17:20:15.156 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:20:15.157 INFO:teuthology.orchestra.run.smithi159.stdout:Total 113 kB/s | 109 kB 00:00 2024-03-25T17:20:15.157 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:20:15.161 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:20:15.161 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:20:15.198 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:20:15.199 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:20:15.314 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:20:15.422 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:20:15.807 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:20:16.096 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:20:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:16.097 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:20:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:16.097 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:16.097 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:20:16.283 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:20:16.849 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:08:29 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:20:16.997 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:20:16.997 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:16.997 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:20:16.997 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:16.997 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:20:16.998 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:20:16.998 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:16.998 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:20:16.998 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:16.998 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:20:16.998 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:16.998 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 5.4 M 2024-03-25T17:20:16.999 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 25 M 2024-03-25T17:20:16.999 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:20:33.057 INFO:teuthology.orchestra.run.smithi159.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 341 kB/s | 5.4 MB 00:16 2024-03-25T17:20:33.058 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:20:33.058 INFO:teuthology.orchestra.run.smithi159.stdout:Total 341 kB/s | 5.4 MB 00:16 2024-03-25T17:20:33.058 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:20:33.062 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:20:33.062 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:20:33.133 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:20:33.133 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:20:33.798 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:20:33.905 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:20:34.409 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:20:34.685 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:20:34.685 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:34.685 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:20:34.685 INFO:teuthology.orchestra.run.smithi159.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:34.685 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:34.686 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:20:34.870 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librados2-debuginfo 2024-03-25T17:20:35.437 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:08:48 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:20:35.585 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:20:35.586 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:35.586 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:20:35.586 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:35.586 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 44 M 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 253 M 2024-03-25T17:20:35.587 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:20:59.951 INFO:teuthology.orchestra.run.smithi134.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 398 kB/s | 52 MB 02:13 2024-03-25T17:20:59.951 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:20:59.952 INFO:teuthology.orchestra.run.smithi134.stdout:Total 398 kB/s | 52 MB 02:13 2024-03-25T17:20:59.952 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:20:59.955 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:20:59.956 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:21:00.299 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:21:00.300 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:21:06.270 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:21:06.365 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:21:08.490 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:21:08.760 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:21:08.760 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:08.760 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:21:08.761 INFO:teuthology.orchestra.run.smithi134.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:08.761 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:08.761 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:21:08.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:21:09.534 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:09:21 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:21:09.684 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:21:09.685 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:09.685 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:21:09.685 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:09.685 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:21:09.685 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 615 k 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.1 M 2024-03-25T17:21:09.686 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:21:12.935 INFO:teuthology.orchestra.run.smithi134.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 189 kB/s | 615 kB 00:03 2024-03-25T17:21:12.936 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:12.936 INFO:teuthology.orchestra.run.smithi134.stdout:Total 189 kB/s | 615 kB 00:03 2024-03-25T17:21:12.937 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:21:12.941 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:21:12.941 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:21:12.982 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:21:12.982 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:21:13.147 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:21:13.243 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:21:13.657 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:21:13.926 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:21:13.926 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:13.926 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:21:13.927 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:13.927 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:13.927 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:21:14.136 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:21:14.701 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:09:26 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:21:14.852 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:21:14.852 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:14.852 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:21:14.852 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 771 k 2024-03-25T17:21:14.853 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.7 M 2024-03-25T17:21:14.854 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:21:18.071 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 240 kB/s | 771 kB 00:03 2024-03-25T17:21:18.072 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:18.072 INFO:teuthology.orchestra.run.smithi134.stdout:Total 239 kB/s | 771 kB 00:03 2024-03-25T17:21:18.073 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:21:18.077 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:21:18.077 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:21:18.118 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:21:18.119 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:21:18.305 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:21:18.422 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:21:18.812 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:21:19.093 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:21:19.093 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:19.093 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:21:19.093 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:19.093 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:19.094 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:21:19.326 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:21:19.890 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:09:31 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:21:20.040 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:21:20.040 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.1 M 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 4.3 M 2024-03-25T17:21:20.041 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:21:23.049 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 383 kB/s | 1.1 MB 00:03 2024-03-25T17:21:23.050 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:23.050 INFO:teuthology.orchestra.run.smithi134.stdout:Total 382 kB/s | 1.1 MB 00:03 2024-03-25T17:21:23.050 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:21:23.054 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:21:23.054 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:21:23.099 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:21:23.099 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:21:23.319 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:21:23.424 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:21:23.813 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:21:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:21:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:24.084 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:21:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:24.084 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:24.084 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:21:24.308 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:21:24.869 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:09:36 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:21:25.019 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:21:25.019 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:25.019 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:21:25.019 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:25.020 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 337 k 2024-03-25T17:21:25.021 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 1.0 M 2024-03-25T17:21:25.021 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:21:26.418 INFO:teuthology.orchestra.run.smithi134.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 241 kB/s | 337 kB 00:01 2024-03-25T17:21:26.419 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:26.420 INFO:teuthology.orchestra.run.smithi134.stdout:Total 241 kB/s | 337 kB 00:01 2024-03-25T17:21:26.420 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:21:26.424 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:21:26.424 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:21:26.464 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:21:26.464 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:21:26.604 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:21:26.710 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:21:27.061 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:21:27.340 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:21:27.340 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:27.340 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:21:27.340 INFO:teuthology.orchestra.run.smithi134.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:27.340 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:27.340 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:21:27.575 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:21:28.133 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:09:40 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:21:28.282 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:21:28.283 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:28.284 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 509 k 2024-03-25T17:21:28.284 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 2.5 M 2024-03-25T17:21:28.284 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:21:30.405 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 240 kB/s | 509 kB 00:02 2024-03-25T17:21:30.405 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:21:30.405 INFO:teuthology.orchestra.run.smithi134.stdout:Total 240 kB/s | 509 kB 00:02 2024-03-25T17:21:30.406 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:21:30.410 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:21:30.410 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:21:30.450 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:21:30.450 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:21:30.620 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:21:30.713 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:21:31.096 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:21:31.366 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:21:31.366 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:31.366 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:21:31.366 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:21:31.367 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:31.367 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:21:31.623 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:21:32.183 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:09:44 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:21:32.335 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:21:32.335 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:32.335 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:21:32.335 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:32.335 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:21:32.335 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:21:32.335 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:32.336 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:21:32.336 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:21:32.336 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:21:32.336 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:21:32.336 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 52 M 2024-03-25T17:21:32.336 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 298 M 2024-03-25T17:21:32.336 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:23:18.856 INFO:teuthology.orchestra.run.smithi159.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 278 kB/s | 44 MB 02:43 2024-03-25T17:23:18.857 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:23:18.857 INFO:teuthology.orchestra.run.smithi159.stdout:Total 278 kB/s | 44 MB 02:43 2024-03-25T17:23:18.857 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:23:18.860 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:23:18.860 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:23:19.159 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:23:19.159 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:23:24.242 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:23:24.354 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:23:26.261 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:23:26.525 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:23:26.525 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:23:26.525 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:23:26.525 INFO:teuthology.orchestra.run.smithi159.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:23:26.525 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:23:26.525 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:23:26.705 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:23:27.271 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:11:40 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:23:27.420 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:23:27.421 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:23:27.422 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:23:27.423 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 52 M 2024-03-25T17:23:27.423 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 308 M 2024-03-25T17:23:27.423 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:24:56.321 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 259 kB/s | 52 MB 03:23 2024-03-25T17:24:56.322 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:24:56.322 INFO:teuthology.orchestra.run.smithi134.stdout:Total 259 kB/s | 52 MB 03:23 2024-03-25T17:24:56.323 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:24:56.326 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:24:56.326 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:24:56.664 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:24:56.665 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:25:02.569 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:25:02.794 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:25:04.811 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:25:05.081 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:25:05.081 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:25:05.082 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:25:05.082 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:05.082 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:25:05.082 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:25:05.286 DEBUG:teuthology.orchestra.run.smithi134:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:25:05.854 INFO:teuthology.orchestra.run.smithi134.stdout:Last metadata expiration check: 1:13:17 ago on Mon 25 Mar 2024 04:11:48 PM UTC. 2024-03-25T17:25:06.004 INFO:teuthology.orchestra.run.smithi134.stdout:Dependencies resolved. 2024-03-25T17:25:06.004 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:25:06.004 INFO:teuthology.orchestra.run.smithi134.stdout: Package Arch Version Repo Size 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout:Installing: 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction Summary 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout:================================================================================ 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout:Install 1 Package 2024-03-25T17:25:06.005 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:25:06.006 INFO:teuthology.orchestra.run.smithi134.stdout:Total download size: 1.3 M 2024-03-25T17:25:06.006 INFO:teuthology.orchestra.run.smithi134.stdout:Installed size: 5.5 M 2024-03-25T17:25:06.006 INFO:teuthology.orchestra.run.smithi134.stdout:Downloading Packages: 2024-03-25T17:25:13.431 INFO:teuthology.orchestra.run.smithi134.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 173 kB/s | 1.3 MB 00:07 2024-03-25T17:25:13.432 INFO:teuthology.orchestra.run.smithi134.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:13.432 INFO:teuthology.orchestra.run.smithi134.stdout:Total 173 kB/s | 1.3 MB 00:07 2024-03-25T17:25:13.432 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction check 2024-03-25T17:25:13.436 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction check succeeded. 2024-03-25T17:25:13.436 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction test 2024-03-25T17:25:13.480 INFO:teuthology.orchestra.run.smithi134.stdout:Transaction test succeeded. 2024-03-25T17:25:13.481 INFO:teuthology.orchestra.run.smithi134.stdout:Running transaction 2024-03-25T17:25:13.735 INFO:teuthology.orchestra.run.smithi134.stdout: Preparing : 1/1 2024-03-25T17:25:13.879 INFO:teuthology.orchestra.run.smithi134.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:14.332 INFO:teuthology.orchestra.run.smithi134.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:14.614 INFO:teuthology.orchestra.run.smithi134.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:14.615 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:25:14.615 INFO:teuthology.orchestra.run.smithi134.stdout:Installed: 2024-03-25T17:25:14.615 INFO:teuthology.orchestra.run.smithi134.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:14.615 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:25:14.615 INFO:teuthology.orchestra.run.smithi134.stdout:Complete! 2024-03-25T17:25:14.829 DEBUG:teuthology.parallel:result is None 2024-03-25T17:25:48.390 INFO:teuthology.orchestra.run.smithi159.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 377 kB/s | 52 MB 02:20 2024-03-25T17:25:48.391 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:25:48.391 INFO:teuthology.orchestra.run.smithi159.stdout:Total 377 kB/s | 52 MB 02:20 2024-03-25T17:25:48.391 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:25:48.395 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:25:48.395 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:25:48.740 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:25:48.740 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:25:54.739 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:25:54.856 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:57.086 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:57.352 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:25:57.352 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:25:57.352 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:25:57.353 INFO:teuthology.orchestra.run.smithi159.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:25:57.353 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:25:57.353 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:25:57.577 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:25:58.140 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:14:11 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:25:58.290 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:25:58.291 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:25:58.292 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 615 k 2024-03-25T17:25:58.292 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.1 M 2024-03-25T17:25:58.292 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:26:00.809 INFO:teuthology.orchestra.run.smithi159.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 244 kB/s | 615 kB 00:02 2024-03-25T17:26:00.810 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:26:00.810 INFO:teuthology.orchestra.run.smithi159.stdout:Total 244 kB/s | 615 kB 00:02 2024-03-25T17:26:00.810 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:26:00.814 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:26:00.814 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:26:00.856 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:26:00.857 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:26:01.028 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:26:01.135 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:26:01.532 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:26:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:26:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:01.810 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:26:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:26:01.810 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:01.810 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:26:01.993 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:26:02.558 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:14:15 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:26:02.710 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:26:02.710 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:02.710 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:26:02.711 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:02.712 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 771 k 2024-03-25T17:26:02.712 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.7 M 2024-03-25T17:26:02.712 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:26:06.812 INFO:teuthology.orchestra.run.smithi159.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 188 kB/s | 771 kB 00:04 2024-03-25T17:26:06.813 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:26:06.813 INFO:teuthology.orchestra.run.smithi159.stdout:Total 188 kB/s | 771 kB 00:04 2024-03-25T17:26:06.813 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:26:06.817 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:26:06.817 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:26:06.858 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:26:06.858 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:26:07.045 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:26:07.153 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:26:07.517 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:26:07.800 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:26:07.800 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:07.800 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:26:07.801 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:26:07.801 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:07.801 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:26:08.019 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:26:08.586 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:14:21 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:26:08.735 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:26:08.735 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:08.735 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:26:08.735 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:08.735 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:26:08.735 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:26:08.736 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:08.736 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:26:08.736 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:08.736 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:26:08.736 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:08.736 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.1 M 2024-03-25T17:26:08.737 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 4.3 M 2024-03-25T17:26:08.737 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:26:14.273 INFO:teuthology.orchestra.run.smithi159.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 208 kB/s | 1.1 MB 00:05 2024-03-25T17:26:14.274 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:26:14.274 INFO:teuthology.orchestra.run.smithi159.stdout:Total 208 kB/s | 1.1 MB 00:05 2024-03-25T17:26:14.274 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:26:14.278 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:26:14.278 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:26:14.321 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:26:14.322 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:26:14.548 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:26:14.647 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:26:15.057 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:26:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:26:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:15.341 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:26:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:26:15.341 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:15.341 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:26:15.532 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:26:16.097 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:14:29 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:16.248 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:26:16.249 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:16.249 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:26:16.249 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:16.249 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 337 k 2024-03-25T17:26:16.249 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 1.0 M 2024-03-25T17:26:16.249 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:26:18.192 INFO:teuthology.orchestra.run.smithi159.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 174 kB/s | 337 kB 00:01 2024-03-25T17:26:18.193 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:26:18.193 INFO:teuthology.orchestra.run.smithi159.stdout:Total 173 kB/s | 337 kB 00:01 2024-03-25T17:26:18.193 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:26:18.197 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:26:18.197 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:26:18.238 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:26:18.239 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:26:18.382 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:26:18.491 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:26:18.898 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:26:19.174 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:26:19.174 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:19.174 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:26:19.174 INFO:teuthology.orchestra.run.smithi159.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:26:19.174 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:19.174 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:26:19.384 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:26:19.948 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:14:32 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:26:20.098 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:26:20.098 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:20.098 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:26:20.098 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:20.098 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:26:20.098 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:26:20.098 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:20.099 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:26:20.099 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:20.099 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:26:20.099 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:20.099 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 509 k 2024-03-25T17:26:20.099 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 2.5 M 2024-03-25T17:26:20.099 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:26:23.578 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 146 kB/s | 509 kB 00:03 2024-03-25T17:26:23.579 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:26:23.580 INFO:teuthology.orchestra.run.smithi159.stdout:Total 146 kB/s | 509 kB 00:03 2024-03-25T17:26:23.580 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:26:23.583 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:26:23.584 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:26:23.622 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:26:23.622 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:26:23.784 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:26:23.894 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:24.273 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:24.555 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:26:24.555 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:24.555 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:26:24.555 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:26:24.555 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:24.555 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:26:24.749 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:26:25.317 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:14:38 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:26:25.467 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 52 M 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 298 M 2024-03-25T17:26:25.468 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:28:04.161 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 535 kB/s | 52 MB 01:38 2024-03-25T17:28:04.161 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:04.162 INFO:teuthology.orchestra.run.smithi159.stdout:Total 535 kB/s | 52 MB 01:38 2024-03-25T17:28:04.162 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:28:04.165 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:28:04.165 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:28:04.500 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:28:04.500 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:28:10.426 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:28:10.616 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:28:12.690 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:28:12.972 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:28:12.972 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:12.972 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:28:12.972 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:28:12.972 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:12.972 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:28:13.205 DEBUG:teuthology.orchestra.run.smithi159:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T17:28:13.777 INFO:teuthology.orchestra.run.smithi159.stdout:Last metadata expiration check: 1:16:26 ago on Mon 25 Mar 2024 04:11:47 PM UTC. 2024-03-25T17:28:13.927 INFO:teuthology.orchestra.run.smithi159.stdout:Dependencies resolved. 2024-03-25T17:28:13.928 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout: Package Arch Version Repo Size 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:Installing: 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction Summary 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:================================================================================ 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout:Install 1 Package 2024-03-25T17:28:13.929 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Total download size: 1.3 M 2024-03-25T17:28:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Installed size: 5.5 M 2024-03-25T17:28:13.930 INFO:teuthology.orchestra.run.smithi159.stdout:Downloading Packages: 2024-03-25T17:28:17.562 INFO:teuthology.orchestra.run.smithi159.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 353 kB/s | 1.3 MB 00:03 2024-03-25T17:28:17.562 INFO:teuthology.orchestra.run.smithi159.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:17.563 INFO:teuthology.orchestra.run.smithi159.stdout:Total 353 kB/s | 1.3 MB 00:03 2024-03-25T17:28:17.563 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction check 2024-03-25T17:28:17.567 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction check succeeded. 2024-03-25T17:28:17.567 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction test 2024-03-25T17:28:17.611 INFO:teuthology.orchestra.run.smithi159.stdout:Transaction test succeeded. 2024-03-25T17:28:17.611 INFO:teuthology.orchestra.run.smithi159.stdout:Running transaction 2024-03-25T17:28:17.845 INFO:teuthology.orchestra.run.smithi159.stdout: Preparing : 1/1 2024-03-25T17:28:17.953 INFO:teuthology.orchestra.run.smithi159.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:28:18.344 INFO:teuthology.orchestra.run.smithi159.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:28:18.637 INFO:teuthology.orchestra.run.smithi159.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:28:18.637 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:18.637 INFO:teuthology.orchestra.run.smithi159.stdout:Installed: 2024-03-25T17:28:18.637 INFO:teuthology.orchestra.run.smithi159.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:28:18.638 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:18.638 INFO:teuthology.orchestra.run.smithi159.stdout:Complete! 2024-03-25T17:28:18.827 DEBUG:teuthology.parallel:result is None 2024-03-25T17:28:18.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:28:18.968 DEBUG:teuthology.orchestra.run.smithi134:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:28:18.993 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:28:18.994 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:28:18.994 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:28:18.996 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:28:19.206 DEBUG:teuthology.orchestra.run.smithi159:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T17:28:19.230 INFO:teuthology.orchestra.run.smithi159.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T17:28:19.231 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T17:28:19.231 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T17:28:19.232 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T17:28:19.232 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:19.232 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:28:19.264 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:19.264 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T17:28:19.298 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T17:28:19.298 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:19.298 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:28:19.335 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:28:19.406 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:19.406 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T17:28:19.435 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T17:28:19.502 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T17:28:19.502 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:19.502 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:28:19.533 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:28:19.603 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:19.603 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T17:28:19.630 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T17:28:19.697 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T17:28:19.697 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:19.697 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:28:19.726 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:28:19.794 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:19.795 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T17:28:19.821 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T17:28:19.888 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T17:28:19.961 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T17:28:19.961 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 777 /var/log/ceph 2024-03-25T17:28:19.963 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 777 /var/log/ceph 2024-03-25T17:28:19.990 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T17:28:19.990 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:28:20.033 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T17:28:20.060 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T17:28:20.060 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:28:20.103 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T17:28:20.148 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T17:28:20.148 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T17:28:20.148 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621243', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi134.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpd+Wy2Iyel6QblPc3uH35qdQGkFJ3LR0wpRQxPwEiG4A/yQjQmul8p7bjNIlRnJ5eUtXz3mw7mevtg6b9gLDA=', 'smithi159.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNoDZvHzI5Yx7J7zf/elHmBDSn0RqC4PyojPaeAHlb5s3UwoIlI+7Q97xPwEovjZNdacoFeMV9QOEgynzu+UpP4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T17:28:20.149 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:28:20.172 DEBUG:teuthology.orchestra.run.smithi159:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T17:28:20.186 DEBUG:teuthology.orchestra.run.smithi134:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:28:20.228 DEBUG:teuthology.orchestra.run.smithi159:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T17:28:20.254 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:20.254 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:28:20.309 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:28:20.309 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_1 2024-03-25T17:28:20.364 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:28:20.364 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:20.364 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T17:28:20.364 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:20.364 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:20.364 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 16:14:09.705267873 +0000 2024-03-25T17:28:20.364 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 16:11:03.154785847 +0000 2024-03-25T17:28:20.365 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 16:11:03.154785847 +0000 2024-03-25T17:28:20.365 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 16:11:03.154785847 +0000 2024-03-25T17:28:20.365 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:28:20.430 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T17:28:20.430 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T17:28:20.430 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00025912 s, 2.0 MB/s 2024-03-25T17:28:20.431 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:28:20.487 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_2 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 16:14:09.705267873 +0000 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 16:11:03.502777662 +0000 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 16:11:03.502777662 +0000 2024-03-25T17:28:20.542 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 16:11:03.502777662 +0000 2024-03-25T17:28:20.542 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:28:20.607 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T17:28:20.608 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T17:28:20.608 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.000269621 s, 1.9 MB/s 2024-03-25T17:28:20.609 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:28:20.664 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_3 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 16:14:09.706267850 +0000 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 16:11:03.838769761 +0000 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 16:11:03.838769761 +0000 2024-03-25T17:28:20.719 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 16:11:03.838769761 +0000 2024-03-25T17:28:20.720 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:28:20.784 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T17:28:20.785 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T17:28:20.785 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00025995 s, 2.0 MB/s 2024-03-25T17:28:20.786 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:28:20.841 DEBUG:teuthology.orchestra.run.smithi134:> stat /dev/vg_nvme/lv_4 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Access: 2024-03-25 16:14:09.706267850 +0000 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Modify: 2024-03-25 16:11:04.195761365 +0000 2024-03-25T17:28:20.896 INFO:teuthology.orchestra.run.smithi134.stdout:Change: 2024-03-25 16:11:04.195761365 +0000 2024-03-25T17:28:20.897 INFO:teuthology.orchestra.run.smithi134.stdout: Birth: 2024-03-25 16:11:04.195761365 +0000 2024-03-25T17:28:20.897 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:28:20.961 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records in 2024-03-25T17:28:20.962 INFO:teuthology.orchestra.run.smithi134.stderr:1+0 records out 2024-03-25T17:28:20.962 INFO:teuthology.orchestra.run.smithi134.stderr:512 bytes copied, 0.00020806 s, 2.5 MB/s 2024-03-25T17:28:20.963 DEBUG:teuthology.orchestra.run.smithi134:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:28:21.018 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:21.019 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:21.019 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T17:28:21.037 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T17:28:21.037 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_1 2024-03-25T17:28:21.095 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T17:28:21.095 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:21.095 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T17:28:21.095 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:21.095 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:21.095 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 16:13:59.798943287 +0000 2024-03-25T17:28:21.095 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 16:10:32.779888614 +0000 2024-03-25T17:28:21.096 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 16:10:32.779888614 +0000 2024-03-25T17:28:21.096 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 16:10:32.779888614 +0000 2024-03-25T17:28:21.096 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T17:28:21.165 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T17:28:21.166 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T17:28:21.166 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000248123 s, 2.1 MB/s 2024-03-25T17:28:21.167 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T17:28:21.226 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_2 2024-03-25T17:28:21.283 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T17:28:21.283 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:21.284 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T17:28:21.284 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:21.284 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:21.284 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 16:13:59.798943287 +0000 2024-03-25T17:28:21.284 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 16:10:33.101882541 +0000 2024-03-25T17:28:21.284 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 16:10:33.101882541 +0000 2024-03-25T17:28:21.284 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 16:10:33.101882541 +0000 2024-03-25T17:28:21.284 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T17:28:21.352 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T17:28:21.352 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T17:28:21.353 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.00020683 s, 2.5 MB/s 2024-03-25T17:28:21.353 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T17:28:21.410 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_3 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 16:13:59.799943268 +0000 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 16:10:33.437876204 +0000 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 16:10:33.437876204 +0000 2024-03-25T17:28:21.466 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 16:10:33.437876204 +0000 2024-03-25T17:28:21.467 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T17:28:21.532 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T17:28:21.533 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T17:28:21.533 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000207456 s, 2.5 MB/s 2024-03-25T17:28:21.534 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T17:28:21.590 DEBUG:teuthology.orchestra.run.smithi159:> stat /dev/vg_nvme/lv_4 2024-03-25T17:28:21.645 INFO:teuthology.orchestra.run.smithi159.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout:Access: 2024-03-25 16:13:59.799943268 +0000 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout:Modify: 2024-03-25 16:10:33.781869715 +0000 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout:Change: 2024-03-25 16:10:33.781869715 +0000 2024-03-25T17:28:21.646 INFO:teuthology.orchestra.run.smithi159.stdout: Birth: 2024-03-25 16:10:33.781869715 +0000 2024-03-25T17:28:21.646 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T17:28:21.711 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records in 2024-03-25T17:28:21.711 INFO:teuthology.orchestra.run.smithi159.stderr:1+0 records out 2024-03-25T17:28:21.711 INFO:teuthology.orchestra.run.smithi159.stderr:512 bytes copied, 0.000181866 s, 2.8 MB/s 2024-03-25T17:28:21.712 DEBUG:teuthology.orchestra.run.smithi159:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T17:28:21.768 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:21.768 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:28:21.768 INFO:tasks.ceph:Generating config... 2024-03-25T17:28:21.769 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T17:28:21.769 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T17:28:21.769 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T17:28:21.769 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T17:28:21.769 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T17:28:21.769 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T17:28:21.769 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T17:28:21.770 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T17:28:21.770 INFO:tasks.ceph:[global] ms type = async 2024-03-25T17:28:21.770 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T17:28:21.770 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T17:28:21.771 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T17:28:21.771 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T17:28:21.772 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T17:28:21.773 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T17:28:21.773 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:21.826 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T17:28:21.828 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T17:28:21.873 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:28:21.900 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.134:6789]'), ('mon.c', '[v1:172.21.15.134:6790]'), ('mon.b', '[v1:172.21.15.159:6789]')] 2024-03-25T17:28:21.900 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.134:6789],[v1:172.21.15.134:6790],[v1:172.21.15.159:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T17:28:21.901 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:21.901 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:28:21.955 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.134:6789]' --addv c '[v1:172.21.15.134:6790]' --addv b '[v1:172.21.15.159:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: generated fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:setting min_mon_release = quincy 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:epoch 0 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:28:22.040 INFO:teuthology.orchestra.run.smithi134.stdout:min_mon_release 17 (quincy) 2024-03-25T17:28:22.041 INFO:teuthology.orchestra.run.smithi134.stdout:election_strategy: 3 2024-03-25T17:28:22.041 INFO:teuthology.orchestra.run.smithi134.stdout:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:28:22.041 INFO:teuthology.orchestra.run.smithi134.stdout:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:28:22.041 INFO:teuthology.orchestra.run.smithi134.stdout:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:28:22.041 INFO:teuthology.orchestra.run.smithi134.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T17:28:22.042 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T17:28:22.056 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d322d3f4-cbbc-46c7-9a8d-3205a340418d... 2024-03-25T17:28:22.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:28:22.098 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout:[global] 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout: chdir = "" 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout: auth supported = cephx 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout: filestore xattr use omap = true 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.146 INFO:teuthology.orchestra.run.smithi134.stdout: mon clock drift allowed = 1.000 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: osd crush chooseleaf type = 0 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: auth debug = true 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on old message = true 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: ms die on bug = true 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: mon pg warn max object skew = 0 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.147 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default size = 2 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow primary affinity = true 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd allow pg remap = true 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on no sortbitwise = false 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on too few osds = false 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: mon_allow_pool_size_one = true 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:28:22.148 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: osd default data pool replay window = 5 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: mon allow pool delete = true 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: mon cluster log file level = debug 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: debug asserts on shutdown = true 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: mon health detail to clog = false 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: mon host = "[v1:172.21.15.134:6789],[v1:172.21.15.134:6790],[v1:172.21.15.159:6789]" 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: mon client directed command retry = 5 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: mon election default strategy = 3 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: ms bind msgr2 = False 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: ms inject socket failures = 5000 2024-03-25T17:28:22.149 INFO:teuthology.orchestra.run.smithi134.stdout: ms type = async 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd heartbeat grace = 80 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: fsid = d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout:[osd] 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd journal size = 100 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub load threshold = 5.0 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd scrub max interval = 600 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd recover clone overlap = true 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd recovery max chunk = 1048576 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug shutdown = true 2024-03-25T17:28:22.150 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug op order = true 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify stray on activate = true 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug trim objects = true 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: osd open classes on start = true 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug pg log writeout = true 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: osd map max advance = 10 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: journal zero on create = true 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: filestore apply finisher threads = 3 2024-03-25T17:28:22.151 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: bdev debug aio = true 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug misdirected ops = true 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: bdev async discard = True 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: bdev enable discard = True 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore allocator = stupid 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore block size = 96636764160 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore fsck on mount = True 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: bluestore zero block detection = True 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluefs = 1 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: debug bluestore = 1 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: debug monc = 20 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: debug osd = 20 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: debug rocksdb = 10 2024-03-25T17:28:22.152 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd full ratio = 0.9 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd class default list = * 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd class load list = * 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd client watch timeout = 120 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify cached snaps = True 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd debug verify missing on start = True 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd fast shutdown = False 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd max markdown count = 1000 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd mclock override recovery settings = True 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd objectstore = bluestore 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd op queue = debug_random 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd op queue cut off = debug_random 2024-03-25T17:28:22.153 INFO:teuthology.orchestra.run.smithi134.stdout: osd shutdown pgref assert = True 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr] 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug mgr = 20 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: mgr/telemetry/nag = false 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout:[mon] 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug mon = 20 2024-03-25T17:28:22.154 INFO:teuthology.orchestra.run.smithi134.stdout: debug paxos = 20 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: debug auth = 20 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon data avail warn = 5 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon mgr mkfs grace = 240 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd reporter subtree level = osd 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd prime pg temp = true 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:28:22.155 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: # 1m isn't quite enough 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon min osdmap epochs = 50 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon osd crush smoke test = False 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune interval = 2 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune min = 15 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon scrub interval = 300 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: mon warn on pool no app = False 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: paxos service trim min = 10 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout:[client] 2024-03-25T17:28:22.156 INFO:teuthology.orchestra.run.smithi134.stdout: rgw cache enabled = true 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable ops log = true 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: rgw enable usage log = true 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: debug monc = 20 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: debug ms = 1 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: debug objecter = 20 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout: debug rados = 20 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.a] 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.c] 2024-03-25T17:28:22.157 INFO:teuthology.orchestra.run.smithi134.stdout:[mon.b] 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout:[global] 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: chdir = "" 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: auth supported = cephx 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: filestore xattr use omap = true 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: mon clock drift allowed = 1.000 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: osd crush chooseleaf type = 0 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: auth debug = true 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.158 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on old message = true 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: ms die on bug = true 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: mon pg warn max object skew = 0 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default size = 2 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow primary affinity = true 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd allow pg remap = true 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on legacy crush tunables = false 2024-03-25T17:28:22.159 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on crush straw calc version zero = false 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on no sortbitwise = false 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on osd down out interval zero = false 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on too few osds = false 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon_allow_pool_size_one = true 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: osd default data pool replay window = 5 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon allow pool delete = true 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon cluster log file level = debug 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: debug asserts on shutdown = true 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon health detail to clog = false 2024-03-25T17:28:22.160 INFO:teuthology.orchestra.run.smithi159.stdout: mon host = "[v1:172.21.15.134:6789],[v1:172.21.15.134:6790],[v1:172.21.15.159:6789]" 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: mon client directed command retry = 5 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: mon election default strategy = 3 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: ms bind msgr2 = False 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: ms inject socket failures = 5000 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: ms type = async 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: osd heartbeat grace = 80 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: fsid = d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout:[osd] 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: osd journal size = 100 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub load threshold = 5.0 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: osd scrub max interval = 600 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock profile = high_recovery_ops 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.161 INFO:teuthology.orchestra.run.smithi159.stdout: osd recover clone overlap = true 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd recovery max chunk = 1048576 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug shutdown = true 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug op order = true 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify stray on activate = true 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug trim objects = true 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd open classes on start = true 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug pg log writeout = true 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd deep scrub update digest min age = 30 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: osd map max advance = 10 2024-03-25T17:28:22.162 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: journal zero on create = true 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: filestore ondisk finisher threads = 3 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: filestore apply finisher threads = 3 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: bdev debug aio = true 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug misdirected ops = true 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: bdev async discard = True 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: bdev enable discard = True 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore allocator = stupid 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore block size = 96636764160 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore fsck on mount = True 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: bluestore zero block detection = True 2024-03-25T17:28:22.163 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluefs = 1 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: debug bluestore = 1 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: debug monc = 20 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: debug osd = 20 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: debug rocksdb = 10 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd full ratio = 0.9 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd nearfull ratio = 0.8 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: osd blocked scrub grace period = 3600 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: osd class default list = * 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: osd class load list = * 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: osd client watch timeout = 120 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify cached snaps = True 2024-03-25T17:28:22.164 INFO:teuthology.orchestra.run.smithi159.stdout: osd debug verify missing on start = True 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd failsafe full ratio = 0.95 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd fast shutdown = False 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd max markdown count = 1000 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd mclock override recovery settings = True 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd objectstore = bluestore 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd op queue = debug_random 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd op queue cut off = debug_random 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: osd shutdown pgref assert = True 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr] 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: debug mgr = 20 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:22.165 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: mgr/telemetry/nag = false 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout:[mon] 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: debug mon = 20 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: debug paxos = 20 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: debug auth = 20 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: mon data avail warn = 5 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: mon mgr mkfs grace = 240 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min pgs per osd = 4 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd reporter subtree level = osd 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd prime pg temp = true 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: mon reweight min bytes per osd = 10 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.166 INFO:teuthology.orchestra.run.smithi159.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: # 1m isn't quite enough 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon_down_mkfs_grace = 2m 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon_warn_on_filestore_osds = false 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon min osdmap epochs = 50 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon osd crush smoke test = False 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune interval = 2 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune min = 15 2024-03-25T17:28:22.167 INFO:teuthology.orchestra.run.smithi159.stdout: mon osdmap full prune txsize = 2 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: mon scrub interval = 300 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: mon warn on pool no app = False 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: paxos service trim min = 10 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout:[client] 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: rgw cache enabled = true 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable ops log = true 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: rgw enable usage log = true 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: debug monc = 20 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: debug ms = 1 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: debug objecter = 20 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout: debug rados = 20 2024-03-25T17:28:22.168 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.a] 2024-03-25T17:28:22.169 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.c] 2024-03-25T17:28:22.169 INFO:teuthology.orchestra.run.smithi159.stdout:[mon.b] 2024-03-25T17:28:22.170 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T17:28:22.170 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T17:28:22.225 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T17:28:22.225 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:22.225 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T17:28:22.240 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:22.240 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T17:28:22.295 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:28:22.295 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:22.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:28:22.295 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:28:22.374 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:22.374 DEBUG:teuthology.orchestra.run.smithi134:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:22.429 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:28:22.429 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:22.429 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T17:28:22.430 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T17:28:22.469 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:22.469 DEBUG:teuthology.orchestra.run.smithi159:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:22.523 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T17:28:22.523 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T17:28:22.523 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:28:22.578 INFO:teuthology.orchestra.run.smithi134.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T17:28:22.580 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:28:22.644 INFO:teuthology.orchestra.run.smithi159.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T17:28:22.646 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T17:28:22.646 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T17:28:22.646 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T17:28:22.688 INFO:teuthology.orchestra.run.smithi134.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T17:28:22.703 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T17:28:22.746 INFO:teuthology.orchestra.run.smithi159.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T17:28:22.761 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T17:28:22.761 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi134.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi159.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T17:28:22.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:22.787 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:22.788 INFO:tasks.ceph:role: osd.0 2024-03-25T17:28:22.788 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:28:22.788 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:28:22.862 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:22.862 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:22.862 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:22.863 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:22.863 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:22.863 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:22.863 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:23.343 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T17:28:23.345 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T17:28:23.345 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:23.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:23.529 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:23.597 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:23.597 INFO:tasks.ceph:role: osd.1 2024-03-25T17:28:23.597 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:28:23.597 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:28:23.671 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:23.671 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:23.672 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:24.120 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T17:28:24.122 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T17:28:24.122 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:24.210 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:24.242 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:24.310 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:24.310 INFO:tasks.ceph:role: osd.2 2024-03-25T17:28:24.310 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:28:24.311 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:28:24.384 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:24.385 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:24.876 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T17:28:24.879 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T17:28:24.879 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:25.021 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:25.055 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:25.123 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:25.123 INFO:tasks.ceph:role: osd.3 2024-03-25T17:28:25.123 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:28:25.123 DEBUG:teuthology.orchestra.run.smithi134:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:28:25.198 INFO:teuthology.orchestra.run.smithi134.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:25.198 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:25.198 INFO:teuthology.orchestra.run.smithi134.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:25.199 INFO:teuthology.orchestra.run.smithi134.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:25.199 INFO:teuthology.orchestra.run.smithi134.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:25.199 INFO:teuthology.orchestra.run.smithi134.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:25.199 INFO:teuthology.orchestra.run.smithi134.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:25.199 INFO:teuthology.orchestra.run.smithi134.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:25.199 INFO:teuthology.orchestra.run.smithi134.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:25.199 INFO:teuthology.orchestra.run.smithi134.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:25.702 INFO:teuthology.orchestra.run.smithi134.stdout:Discarding blocks...Done. 2024-03-25T17:28:25.704 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi134.front.sepia.ceph.com -o noatime 2024-03-25T17:28:25.704 DEBUG:teuthology.orchestra.run.smithi134:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:25.853 DEBUG:teuthology.orchestra.run.smithi134:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:25.889 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:25.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:25.985+0000 7f6195097640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T17:28:25.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:25.985+0000 7f6195097640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T17:28:25.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:25.985+0000 7f6195097640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:28:25.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:25.985+0000 7f6195097640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:28:25.986 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:25.985+0000 7f6195097640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T17:28:25.987 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:25.986+0000 7f6195097640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T17:28:28.832 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T17:28:28.862 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:28.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:28.950+0000 7f5e1d40f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T17:28:28.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:28.951+0000 7f5e1d40f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T17:28:28.951 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:28.951+0000 7f5e1d40f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:28:28.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:28.951+0000 7f5e1d40f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:28:28.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:28.951+0000 7f5e1d40f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T17:28:28.952 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:28.951+0000 7f5e1d40f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T17:28:31.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T17:28:31.836 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:31.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:31.927+0000 7f46d9e0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T17:28:31.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:31.927+0000 7f46d9e0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T17:28:31.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:31.927+0000 7f46d9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:28:31.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:31.927+0000 7f46d9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:28:31.928 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:31.927+0000 7f46d9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T17:28:31.929 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:31.928+0000 7f46d9e0d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T17:28:34.776 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T17:28:34.807 DEBUG:teuthology.orchestra.run.smithi134:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:34.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:34.897+0000 7f773a29b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T17:28:34.898 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:34.898+0000 7f773a29b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T17:28:34.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:34.898+0000 7f773a29b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:28:34.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:34.898+0000 7f773a29b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:28:34.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:34.898+0000 7f773a29b640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T17:28:34.899 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:34.898+0000 7f773a29b640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T17:28:37.748 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T17:28:37.777 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:37.809 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:37.809 INFO:tasks.ceph:role: osd.4 2024-03-25T17:28:37.809 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:28:37.809 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T17:28:37.887 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:37.887 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:37.887 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:37.888 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:37.888 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:37.888 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:37.888 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:37.888 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:37.888 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:37.888 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:38.385 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T17:28:38.387 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T17:28:38.387 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:38.531 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:38.570 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:38.640 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:38.640 INFO:tasks.ceph:role: osd.5 2024-03-25T17:28:38.640 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:28:38.640 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T17:28:38.716 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:38.716 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:38.716 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:38.717 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:38.717 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:38.717 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:38.717 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:38.717 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:38.717 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:38.717 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:39.171 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T17:28:39.173 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T17:28:39.173 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:39.268 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:39.302 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:39.371 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:39.371 INFO:tasks.ceph:role: osd.6 2024-03-25T17:28:39.371 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:28:39.371 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:39.447 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:39.935 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T17:28:39.937 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T17:28:39.937 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:40.098 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:40.133 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:40.202 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T17:28:40.203 INFO:tasks.ceph:role: osd.7 2024-03-25T17:28:40.203 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:28:40.203 DEBUG:teuthology.orchestra.run.smithi159:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T17:28:40.279 INFO:teuthology.orchestra.run.smithi159.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T17:28:40.279 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T17:28:40.279 INFO:teuthology.orchestra.run.smithi159.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T17:28:40.279 INFO:teuthology.orchestra.run.smithi159.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T17:28:40.279 INFO:teuthology.orchestra.run.smithi159.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T17:28:40.279 INFO:teuthology.orchestra.run.smithi159.stdout: = sunit=0 swidth=0 blks 2024-03-25T17:28:40.279 INFO:teuthology.orchestra.run.smithi159.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T17:28:40.280 INFO:teuthology.orchestra.run.smithi159.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T17:28:40.280 INFO:teuthology.orchestra.run.smithi159.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T17:28:40.280 INFO:teuthology.orchestra.run.smithi159.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T17:28:40.793 INFO:teuthology.orchestra.run.smithi159.stdout:Discarding blocks...Done. 2024-03-25T17:28:40.795 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi159.front.sepia.ceph.com -o noatime 2024-03-25T17:28:40.795 DEBUG:teuthology.orchestra.run.smithi159:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:40.927 DEBUG:teuthology.orchestra.run.smithi159:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:40.965 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:41.061 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:41.060+0000 7f7a7de0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T17:28:41.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:41.061+0000 7f7a7de0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T17:28:41.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:41.061+0000 7f7a7de0c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:28:41.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:41.061+0000 7f7a7de0c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:28:41.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:41.061+0000 7f7a7de0c640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T17:28:41.062 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:41.061+0000 7f7a7de0c640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T17:28:43.904 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T17:28:43.936 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:44.026 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:44.025+0000 7fd0d2201640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T17:28:44.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:44.025+0000 7fd0d2201640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T17:28:44.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:44.026+0000 7fd0d2201640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:28:44.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:44.026+0000 7fd0d2201640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:28:44.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:44.026+0000 7fd0d2201640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T17:28:44.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:44.026+0000 7fd0d2201640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T17:28:46.866 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T17:28:46.897 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:46.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:46.988+0000 7f3594fbd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T17:28:46.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:46.988+0000 7f3594fbd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T17:28:46.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:46.988+0000 7f3594fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:28:46.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:46.988+0000 7f3594fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:28:46.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:46.988+0000 7f3594fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T17:28:46.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:46.989+0000 7f3594fbd640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T17:28:49.842 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T17:28:49.875 DEBUG:teuthology.orchestra.run.smithi159:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:49.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:49.966+0000 7f17d7611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T17:28:49.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:49.967+0000 7f17d7611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T17:28:49.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:49.967+0000 7f17d7611640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:28:49.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:49.967+0000 7f17d7611640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:28:49.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:49.967+0000 7f17d7611640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T17:28:49.968 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:28:49.967+0000 7f17d7611640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T17:28:52.816 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T17:28:52.847 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T17:28:52.847 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:52.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T17:28:52.878 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:52.878 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T17:28:52.948 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:52.948 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T17:28:53.017 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:53.018 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T17:28:53.087 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:53.087 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T17:28:53.156 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:53.156 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T17:28:53.187 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:53.188 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T17:28:53.258 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:53.258 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T17:28:53.328 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:53.328 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T17:28:53.396 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:53.397 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T17:28:53.464 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:28:53.464 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T17:28:53.481 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:28:53.481 DEBUG:teuthology.orchestra.run.smithi159:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T17:28:53.520 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T17:28:53.520 DEBUG:teuthology.orchestra.run.smithi134:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:28:53.522 DEBUG:teuthology.orchestra.run.smithi159:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T17:28:53.563 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.y] 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA2tAFmoA5+IhAAX7sCkxf77HZAeJ45uoq4Lg== 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.0] 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA5tAFmPTvNOhAAhHD0Yaj/c90zJZaLpVlNuw== 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.1] 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA8tAFmhIu8OBAAT8XvKVFbcuVwf9H/YJq7pQ== 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.2] 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA/tAFmMdVWNxAAv2r1YjOZaqzc3upvHPXiig== 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.3] 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBCtAFmEOSRNRAAsRK4ZIaBAsUPkls7ZGrkvg== 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout:[mgr.x] 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA2tAFmMBZsJhAAq6XciPFzJj0mcM3+DkziHg== 2024-03-25T17:28:53.564 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.4] 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBJtAFmktmwAxAA01Eo9HXilMabEWE/9b3ZgQ== 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.5] 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBMtAFmK5SXARAAonRhX2RBIm9w8TwncNZeDw== 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.6] 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBOtAFm7Dj1OhAAwY7ETJfagc9hFZPp2S5I8Q== 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout:[osd.7] 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQBRtAFmevSxORAAXtCrWq7Jpi8xO/7wVRyO0A== 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout:[client.0] 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA2tAFm+vUWKRAADkkzLh8EGal2lcskKCBSYg== 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout:[client.1] 2024-03-25T17:28:53.565 INFO:teuthology.orchestra.run.smithi134.stdout: key = AQA2tAFmYRKBLBAAE35+HcYM8tI30ICB6mnZ7w== 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.y] 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA2tAFmoA5+IhAAX7sCkxf77HZAeJ45uoq4Lg== 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.0] 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA5tAFmPTvNOhAAhHD0Yaj/c90zJZaLpVlNuw== 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.1] 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA8tAFmhIu8OBAAT8XvKVFbcuVwf9H/YJq7pQ== 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.2] 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA/tAFmMdVWNxAAv2r1YjOZaqzc3upvHPXiig== 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.3] 2024-03-25T17:28:53.586 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBCtAFmEOSRNRAAsRK4ZIaBAsUPkls7ZGrkvg== 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout:[mgr.x] 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA2tAFmMBZsJhAAq6XciPFzJj0mcM3+DkziHg== 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.4] 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBJtAFmktmwAxAA01Eo9HXilMabEWE/9b3ZgQ== 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.5] 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBMtAFmK5SXARAAonRhX2RBIm9w8TwncNZeDw== 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.6] 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBOtAFm7Dj1OhAAwY7ETJfagc9hFZPp2S5I8Q== 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout:[osd.7] 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQBRtAFmevSxORAAXtCrWq7Jpi8xO/7wVRyO0A== 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout:[client.0] 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA2tAFm+vUWKRAADkkzLh8EGal2lcskKCBSYg== 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout:[client.1] 2024-03-25T17:28:53.587 INFO:teuthology.orchestra.run.smithi159.stdout: key = AQA2tAFmYRKBLBAAE35+HcYM8tI30ICB6mnZ7w== 2024-03-25T17:28:53.588 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:53.609 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:53.681 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.683 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.735 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.737 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.789 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.791 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.843 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.845 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.896 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:53.898 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T17:28:53.950 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:53.952 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:54.004 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:54.006 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:54.057 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:54.059 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:54.111 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:54.113 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T17:28:54.164 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:54.166 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:54.221 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:54.223 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T17:28:54.277 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T17:28:54.278 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T17:28:54.304 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:54.401 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T17:28:54.428 DEBUG:teuthology.orchestra.run.smithi134:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T17:28:54.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:54.592 DEBUG:teuthology.orchestra.run.smithi134:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T17:28:54.619 DEBUG:teuthology.orchestra.run.smithi159:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T17:28:54.647 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T17:28:54.744 DEBUG:teuthology.orchestra.run.smithi159:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T17:28:54.770 DEBUG:teuthology.orchestra.run.smithi134:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:54.773 DEBUG:teuthology.orchestra.run.smithi159:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T17:28:54.825 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T17:28:54.825 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:28:54.825 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T17:28:54.825 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T17:28:54.828 INFO:tasks.ceph.mon.a:Started 2024-03-25T17:28:54.828 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:28:54.828 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T17:28:54.828 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T17:28:54.830 INFO:tasks.ceph.mon.c:Started 2024-03-25T17:28:54.830 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T17:28:54.830 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T17:28:54.830 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T17:28:54.868 INFO:tasks.ceph.mon.b:Started 2024-03-25T17:28:54.868 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T17:28:54.868 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T17:28:54.868 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T17:28:54.870 INFO:tasks.ceph.mgr.y:Started 2024-03-25T17:28:54.870 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T17:28:54.871 DEBUG:teuthology.orchestra.run.smithi159:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T17:28:54.872 INFO:tasks.ceph.mgr.x:Started 2024-03-25T17:28:54.872 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T17:28:54.873 DEBUG:teuthology.orchestra.run.smithi134:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.973+0000 7f8815732640 10 monclient: get_monmap_and_config 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.973+0000 7f8815732640 10 monclient: build_initial_monmap 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.973+0000 7f8815732640 10 monclient: monmap: 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:54.974821+0000 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:54.974821+0000 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:28:54.974 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:28:54.975 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:28:54.975 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:28:54.975 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:28:54.975 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:28:54.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.974+0000 7f8815732640 1 Processor -- start 2024-03-25T17:28:54.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.974+0000 7f8815732640 1 -- start start 2024-03-25T17:28:54.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.974+0000 7f8815732640 10 monclient: init 2024-03-25T17:28:54.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 10 monclient: _reopen_session rank -1 2024-03-25T17:28:54.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:28:54.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 10 monclient(hunting): picked mon.noname-c con 0x7f88100b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 10 monclient(hunting): picked mon.noname-a con 0x7f8810142a20 addr v1:172.21.15.134:6789/0 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100a9d60 con 0x7f8810142a20 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881013ff10 con 0x7f88100b4700 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100aa1f0 con 0x7f88100a96b0 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f881013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f8815732640 10 monclient(hunting): authenticate will time out at 5288.237948s 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f881013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:54.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:54.975+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f8810142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:28:55.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:55.176+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f881013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:55.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:55.176+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f881013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:55.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:55.176+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f8810142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:28:55.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:55.577+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f881013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:55.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:55.577+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f881013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:55.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:55.577+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f8810142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:28:56.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:56.378+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f881013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:56.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:56.378+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f881013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:56.379 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:56.378+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f8810142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f881013f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f8810142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f881013f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f88100b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f8810142a20 addr v1:172.21.15.134:6789/0 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100b4ae0 con 0x7f8810142a20 2024-03-25T17:28:57.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100a9d60 con 0x7f88100b4700 2024-03-25T17:28:57.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881013ff10 con 0x7f88100a96b0 2024-03-25T17:28:57.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:28:57.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:57.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:57.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:57.975+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:28:58.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:58.176+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:58.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:58.176+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:58.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:58.176+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:28:58.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:58.577+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:58.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:58.577+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:58.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:58.577+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:28:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:59.378+0000 7f880f7fe640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00c760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:28:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:59.378+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:28:59.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:28:59.378+0000 7f880effd640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:29:00.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:00.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:00.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:00.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:00.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00c760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f8810142a20 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f88100b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100aa1f0 con 0x7f8810142a20 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100b4ae0 con 0x7f88100b4700 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100a9d60 con 0x7f88100a96b0 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:29:00.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880effd640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:29:00.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:00.975+0000 7f880f7fe640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:29:01.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:01.176+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:29:01.176 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:01.176+0000 7f880effd640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:29:01.177 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:01.176+0000 7f880f7fe640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:29:01.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:01.577+0000 7f880effd640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:29:01.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:01.577+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:29:01.577 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:01.577+0000 7f880f7fe640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:29:01.824 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:00:06.853 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:29:02.132 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:00:07.160 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:29:02.175 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:00:07.192 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:29:02.176 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:00:07.193 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:29:02.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:02.378+0000 7f880effd640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6790/0 2024-03-25T17:29:02.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:02.378+0000 7f880e7fc640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.134:6789/0 2024-03-25T17:29:02.378 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:02.378+0000 7f880f7fe640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.159:6789/0 2024-03-25T17:29:02.496 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:00:07.512 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:29:02.571 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:00:07.513 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:29:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 1 -- >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:29:03.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 1 -- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 1 -- >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc008e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f88100b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f8810142a20 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f88100a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881013ff10 con 0x7f8810142a20 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100aa1f0 con 0x7f88100b4700 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100b4ae0 con 0x7f88100a96b0 2024-03-25T17:29:03.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.975+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:03.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.996+0000 7f880effd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 0x7f87fc00c750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:50814/0 (socket says 172.21.15.134:50814) 2024-03-25T17:29:03.996 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:03.996+0000 7f880effd640 1 -- 172.21.15.134:0/3594150868 learned_addr learned my addr 172.21.15.134:0/3594150868 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:06.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:06.976 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6790/0 conn(0x7f88100a96b0 legacy=0x7f87fc00ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.159:6789/0 conn(0x7f88100b4700 legacy=0x7f87fc00c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6789/0 conn(0x7f8810142a20 legacy=0x7f87fc010750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc0066e0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc012090 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc015ba0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100a9d60 con 0x7f87fc0066e0 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0107a0 con 0x7f87fc012090 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0109d0 con 0x7f87fc015ba0 2024-03-25T17:29:06.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:06.976+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:09.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:09.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:09.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:09.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:09.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6790/0 conn(0x7f87fc015ba0 legacy=0x7f87fc015f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:09.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.159:6789/0 conn(0x7f87fc012090 legacy=0x7f87fc014480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6789/0 conn(0x7f87fc0066e0 legacy=0x7f87fc008b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc00ffa0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc012090 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc015ba0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc010c00 con 0x7f87fc00ffa0 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0104f0 con 0x7f87fc012090 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc014530 con 0x7f87fc015ba0 2024-03-25T17:29:09.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:09.976+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:12.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:12.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:12.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6790/0 conn(0x7f87fc015ba0 legacy=0x7f87fc008b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.159:6789/0 conn(0x7f87fc012090 legacy=0x7f87fc00d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6789/0 conn(0x7f87fc00ffa0 legacy=0x7f87fc0036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc0146c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc014ac0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc00d740 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc015770 con 0x7f87fc00d740 2024-03-25T17:29:12.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc00db20 con 0x7f87fc0146c0 2024-03-25T17:29:12.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc00dcb0 con 0x7f87fc014ac0 2024-03-25T17:29:12.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:12.976+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:15.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:15.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:15.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:15.977 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6790/0 conn(0x7f87fc014ac0 legacy=0x7f87fc014f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.159:6789/0 conn(0x7f87fc0146c0 legacy=0x7f87fc005de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6789/0 conn(0x7f87fc00d740 legacy=0x7f87fc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc008cb0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc009ce0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc014ac0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc008b20 con 0x7f87fc008cb0 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc00a230 con 0x7f87fc009ce0 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0095d0 con 0x7f87fc014ac0 2024-03-25T17:29:15.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:15.977+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:18.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:18.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:18.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:18.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:18.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6790/0 conn(0x7f87fc014ac0 legacy=0x7f87fc014f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:18.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.159:6789/0 conn(0x7f87fc009ce0 legacy=0x7f87fc017f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:18.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6789/0 conn(0x7f87fc008cb0 legacy=0x7f87fc0036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc0098f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc013b90 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc014ac0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc009760 con 0x7f87fc013b90 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc014050 con 0x7f87fc0098f0 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc0141e0 con 0x7f87fc014ac0 2024-03-25T17:29:18.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:18.977+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:21.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient(hunting): tick 2024-03-25T17:29:21.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T17:29:21.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 1 monclient(hunting): continuing hunt 2024-03-25T17:29:21.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T17:29:21.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6790/0 conn(0x7f87fc014ac0 legacy=0x7f87fc0036c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:21.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.159:6789/0 conn(0x7f87fc0098f0 legacy=0x7f87fc012470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:21.978 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6789/0 conn(0x7f87fc013b90 legacy=0x7f87fc005de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc015070 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc00ce20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc00d2a0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc014ee0 con 0x7f87fc00d2a0 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc011080 con 0x7f87fc00ce20 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87fc013480 con 0x7f87fc015070 2024-03-25T17:29:21.979 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:21.978+0000 7f880cff9640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:22.908 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.908+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53395893 0 0) 0x7f87fc014ee0 con 0x7f87fc00d2a0 2024-03-25T17:29:22.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.908+0000 7f880dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:22.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.908+0000 7f880dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:22.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.908+0000 7f880dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:22.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.908+0000 7f880dffb640 10 monclient(hunting): my global_id is 4101 2024-03-25T17:29:22.909 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.908+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8003520 con 0x7f87fc00d2a0 2024-03-25T17:29:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1335829129 0 0) 0x7f87f8003520 con 0x7f87fc00d2a0 2024-03-25T17:29:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6790/0 conn(0x7f87fc015070 legacy=0x7f87fc00c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.159:6789/0 conn(0x7f87fc00ce20 legacy=0x7f87fc018fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:29:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 1 monclient: found mon.noname-a 2024-03-25T17:29:22.961 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87fc013660 con 0x7f87fc00d2a0 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 10 monclient: _finish_auth 0 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f88080043a0 con 0x7f87fc00d2a0 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f8815732640 5 monclient: authenticate success, global_id 4101 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f8815732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f880dffb640 10 monclient: dump: 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:22.962 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:22.963 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:22.960+0000 7f8815732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:23.031 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8808004570 con 0x7f87fc00d2a0 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f880dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f880dffb640 1 -- 172.21.15.134:0/3594150868 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8808004940 con 0x7f87fc00d2a0 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f8815732640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f8815732640 4 set_mon_vals no callback set 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f880dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f880dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.031+0000 7f880dffb640 10 monclient: dump: 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:23.032 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:23.033 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:23.033 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:23.033 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:23.033 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:23.033 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:23.033 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.032+0000 7f8815732640 10 monclient: shutdown 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.032+0000 7f8815732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.032+0000 7f8815732640 1 -- 172.21.15.134:0/3594150868 >> v1:172.21.15.134:6789/0 conn(0x7f87fc00d2a0 legacy=0x7f87fc01d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.032+0000 7f8815732640 1 -- 172.21.15.134:0/3594150868 shutdown_connections 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.032+0000 7f8815732640 1 -- 172.21.15.134:0/3594150868 >> 172.21.15.134:0/3594150868 conn(0x7f88100668e0 msgr2=0x7f8810066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.032+0000 7f8815732640 1 -- 172.21.15.134:0/3594150868 shutdown_connections 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.032+0000 7f8815732640 1 -- 172.21.15.134:0/3594150868 wait complete. 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 10 monclient: build_initial_monmap 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 10 monclient: monmap: 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:23.036 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 1 librados: starting msgr at 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 1 librados: starting objecter 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 1 Processor -- start 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 1 -- start start 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 1 librados: setting wanted keys 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 1 librados: calling monclient init 2024-03-25T17:29:23.037 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.033+0000 7f8815732640 10 monclient: init 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 10 monclient(hunting): picked mon.noname-b con 0x7f87fc00ce20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 10 monclient(hunting): picked mon.noname-c con 0x7f87fc00d2a0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 10 monclient(hunting): picked mon.noname-a con 0x7f87fc015070 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881006faa0 con 0x7f87fc015070 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88100b16b0 con 0x7f87fc00ce20 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881006fd10 con 0x7f87fc00d2a0 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f8815732640 10 monclient(hunting): authenticate will time out at 5316.296961s 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f880e7fc640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f87fc00d2a0 0x7f8810070e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:52746/0 (socket says 172.21.15.134:52746) 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.034+0000 7f880e7fc640 1 -- 172.21.15.134:0/3409894292 learned_addr learned my addr 172.21.15.134:0/3409894292 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.035+0000 7f87f77fe640 10 client.?.objecter ms_handle_connect 0x7f87fc00d2a0 2024-03-25T17:29:23.038 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.035+0000 7f87f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:23.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.044+0000 7f87f77fe640 10 client.?.objecter ms_handle_connect 0x7f87fc015070 2024-03-25T17:29:23.045 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.044+0000 7f87f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:23.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.053+0000 7f87f77fe640 10 client.?.objecter ms_handle_connect 0x7f87fc00ce20 2024-03-25T17:29:23.054 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.053+0000 7f87f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:23.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.064+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175308801 0 0) 0x7f87fc013660 con 0x7f87fc015070 2024-03-25T17:29:23.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.065+0000 7f87f77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:23.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.065+0000 7f87f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:23.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.065+0000 7f87f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:23.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.065+0000 7f87f77fe640 10 monclient(hunting): my global_id is 4110 2024-03-25T17:29:23.065 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.065+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87fc014ea0 con 0x7f87fc015070 2024-03-25T17:29:23.111 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.111+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2731638198 0 0) 0x7f87fc014ea0 con 0x7f87fc015070 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.111+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87fc013660 con 0x7f87fc015070 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.111+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8808004880 con 0x7f87fc015070 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.111+0000 7f87f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.111+0000 7f87f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.111+0000 7f87f77fe640 10 monclient(hunting): dump: 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:23.112 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:23.113 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:23.113 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.111+0000 7f87f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.174+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017982626 0 0) 0x7f87fc013480 con 0x7f87fc00ce20 2024-03-25T17:29:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.174+0000 7f87f77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.174+0000 7f87f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.174+0000 7f87f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.174+0000 7f87f77fe640 10 monclient(hunting): my global_id is 4105 2024-03-25T17:29:23.175 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.174+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87fc014ea0 con 0x7f87fc00ce20 2024-03-25T17:29:23.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.189+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1855993703 0 0) 0x7f87fc014ea0 con 0x7f87fc00ce20 2024-03-25T17:29:23.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.189+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87fc013480 con 0x7f87fc00ce20 2024-03-25T17:29:23.189 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.189+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f88000040b0 con 0x7f87fc00ce20 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.189+0000 7f87f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.189+0000 7f87f77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.189+0000 7f87f77fe640 10 monclient(hunting): dump: 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:23.190 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1427502455 0 0) 0x7f87fc013480 con 0x7f87fc00ce20 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 >> v1:172.21.15.134:6790/0 conn(0x7f87fc00d2a0 legacy=0x7f8810070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 >> v1:172.21.15.134:6789/0 conn(0x7f87fc015070 legacy=0x7f88101e65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 1 monclient: found mon.b 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881006fff0 con 0x7f87fc00ce20 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:23.727 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f87f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 5 monclient: authenticate success, global_id 4105 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 10 monclient: _renew_subs 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88100b16b0 con 0x7f87fc00ce20 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 10 monclient: _renew_subs 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88100b16b0 con 0x7f87fc00ce20 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.725+0000 7f8815732640 1 librados: init done 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.726+0000 7f8815732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.726+0000 7f8815732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.726+0000 7f8815732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:23.728 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:23.726+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88100b1690 con 0x7f87fc00ce20 2024-03-25T17:29:24.028 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88000044c0 con 0x7f87fc00ce20 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f87f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8800004890 con 0x7f87fc00ce20 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f87f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f87f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f87f77fe640 10 monclient: dump: 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:24.029 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:24.030 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:24.030 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:24.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8800004e40 con 0x7f87fc00ce20 2024-03-25T17:29:24.030 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.028+0000 7f880d7fa640 4 set_mon_vals no callback set 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3094935514 0 0) 0x7f8800006310 con 0x7f87fc00ce20 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 10 client.4105.objecter ms_dispatch 0x7f8810000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 3 client.4105.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 3 client.4105.objecter handle_osd_map decoding full epoch 1 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 20 client.4105.objecter dump_active .. 0 homeless 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f88000066c0 con 0x7f87fc00ce20 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:24.034 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.033+0000 7f87f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:24.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.103+0000 7f8815732640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:29:24.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.103+0000 7f8815732640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:29:24.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.103+0000 7f8815732640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:24.103 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.103+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f88100b1690 con 0x7f87fc00ce20 2024-03-25T17:29:24.298 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:24.298+0000 7f87c240f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:29:24.301 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:24.301+0000 7fed5a02b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T17:29:24.356 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:24.356+0000 7f87c240f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:29:24.356 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:24.356+0000 7fed5a02b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T17:29:24.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.428+0000 7f87f77fe640 1 -- 172.21.15.134:0/3409894292 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f88000053b0 con 0x7f87fc00ce20 2024-03-25T17:29:24.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.428+0000 7f87f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T17:29:24.429 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.428+0000 7f87f77fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T17:29:24.429 INFO:teuthology.orchestra.run.smithi134.stderr:adjusted tunables profile to default 2024-03-25T17:29:24.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 10 librados: watch_flush enter 2024-03-25T17:29:24.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 10 librados: watch_flush exit 2024-03-25T17:29:24.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 20 client.4105.objecter shutdown clearing up homeless session... 2024-03-25T17:29:24.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 10 client.4105.objecter successfully canceled tick 2024-03-25T17:29:24.430 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 10 monclient: shutdown 2024-03-25T17:29:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 >> v1:172.21.15.159:6789/0 conn(0x7f87fc00ce20 legacy=0x7f8810070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 shutdown_connections 2024-03-25T17:29:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 >> 172.21.15.134:0/3409894292 conn(0x7f88100668e0 msgr2=0x7f881013f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 shutdown_connections 2024-03-25T17:29:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 1 -- 172.21.15.134:0/3409894292 wait complete. 2024-03-25T17:29:24.431 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:24.429+0000 7f8815732640 1 librados: shutdown 2024-03-25T17:29:24.442 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T17:29:24.442 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T17:29:24.442 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:29:24.443 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T17:29:24.469 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:29:24.469 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T17:29:24.536 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:29:24.536 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T17:29:24.603 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-03-25T17:29:24.603 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T17:29:24.670 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:29:24.670 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T17:29:24.698 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:29:24.698 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T17:29:24.766 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:29:24.766 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T17:29:24.832 DEBUG:teuthology.orchestra.run.smithi159:> set -ex 2024-03-25T17:29:24.833 DEBUG:teuthology.orchestra.run.smithi159:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T17:29:24.898 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:24.898+0000 7f87c240f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:29:24.899 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new bee90ced-7048-4b40-a256-52b8e0df28b8 0 2024-03-25T17:29:24.901 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:24.900+0000 7fed5a02b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T17:29:24.952 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:24.951+0000 7f87c240f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:29:24.959 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:24.958+0000 7fed5a02b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T17:29:24.997 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:24.997+0000 7f87c240f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:29:25.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:24.999+0000 7f024cbaf640 10 monclient: get_monmap_and_config 2024-03-25T17:29:25.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:24.999+0000 7f024cbaf640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:24.999+0000 7f024cbaf640 10 monclient: monmap: 2024-03-25T17:29:25.000 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:25.000 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.000 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:25.000849+0000 2024-03-25T17:29:25.000 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:25.000849+0000 2024-03-25T17:29:25.001 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.001 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.001 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:25.001 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:25.001 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:25.001 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.000+0000 7f024cbaf640 1 Processor -- start 2024-03-25T17:29:25.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.000+0000 7f024cbaf640 1 -- start start 2024-03-25T17:29:25.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.000+0000 7f024cbaf640 10 monclient: init 2024-03-25T17:29:25.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:25.002 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f02480b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f0248142a20 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f02480a96b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480a9d60 con 0x7f02480b4700 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024813ff10 con 0x7f02480a96b0 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480aa1f0 con 0x7f0248142a20 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f024cbaf640 10 monclient(hunting): authenticate will time out at 5300.323859s 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f023ffff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f0248142a20 0x7f024813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:32778/0 (socket says 172.21.15.159:32778) 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f0246d9d640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f02480a96b0 0x7f024813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35450/0 (socket says 172.21.15.159:35450) 2024-03-25T17:29:25.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.001+0000 7f023ffff640 1 -- 172.21.15.159:0/1993822410 learned_addr learned my addr 172.21.15.159:0/1993822410 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:25.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099598375 0 0) 0x7f024813ff10 con 0x7f02480a96b0 2024-03-25T17:29:25.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): my global_id is 4111 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0234003520 con 0x7f02480a96b0 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536465674 0 0) 0x7f02480a9d60 con 0x7f02480b4700 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 10 monclient(hunting): my global_id is 4116 2024-03-25T17:29:25.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.004+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f024813ff10 con 0x7f02480b4700 2024-03-25T17:29:25.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.005+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3985296386 0 0) 0x7f02480aa1f0 con 0x7f0248142a20 2024-03-25T17:29:25.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.005+0000 7f0245d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.005+0000 7f0245d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.005+0000 7f0245d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.005+0000 7f0245d9b640 10 monclient(hunting): my global_id is 34103 2024-03-25T17:29:25.007 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.005+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02480a9d60 con 0x7f0248142a20 2024-03-25T17:29:25.007 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.006+0000 7fed5a02b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T17:29:25.008 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178126208 0 0) 0x7f0234003520 con 0x7f02480a96b0 2024-03-25T17:29:25.008 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 >> v1:172.21.15.134:6790/0 conn(0x7f0248142a20 legacy=0x7f024813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 >> v1:172.21.15.134:6789/0 conn(0x7f02480b4700 legacy=0x7f0248142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 1 monclient: found mon.noname-c 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02480b4ae0 con 0x7f02480a96b0 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f0245d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f024cbaf640 5 monclient: authenticate success, global_id 4111 2024-03-25T17:29:25.009 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.007+0000 7f024cbaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:25.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.010+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f02400019d0 con 0x7f02480a96b0 2024-03-25T17:29:25.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.010+0000 7f0245d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.010+0000 7f0245d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:25.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.010+0000 7f0245d9b640 10 monclient: dump: 2024-03-25T17:29:25.011 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.010+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0240001c20 con 0x7f02480a96b0 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.010+0000 7f0245d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f0245d9b640 1 -- 172.21.15.159:0/1993822410 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0240001fb0 con 0x7f02480a96b0 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f0245d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f0245d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:25.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f0245d9b640 10 monclient: dump: 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:25.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 4 set_mon_vals no callback set 2024-03-25T17:29:25.015 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 10 monclient: shutdown 2024-03-25T17:29:25.015 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 1 -- 172.21.15.159:0/1993822410 >> v1:172.21.15.159:6789/0 conn(0x7f02480a96b0 legacy=0x7f024813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 1 -- 172.21.15.159:0/1993822410 shutdown_connections 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 1 -- 172.21.15.159:0/1993822410 >> 172.21.15.159:0/1993822410 conn(0x7f02480668e0 msgr2=0x7f0248066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 1 -- 172.21.15.159:0/1993822410 shutdown_connections 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.011+0000 7f024cbaf640 1 -- 172.21.15.159:0/1993822410 wait complete. 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.012+0000 7f024cbaf640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.012+0000 7f024cbaf640 10 monclient: monmap: 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:25.016 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 librados: starting msgr at 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 librados: starting objecter 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 Processor -- start 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 -- start start 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 librados: setting wanted keys 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 librados: calling monclient init 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient: init 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f02480a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f02480b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f0248142a20 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:25.017 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480ae060 con 0x7f0248142a20 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02480ae2b0 con 0x7f02480b4700 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02481d7520 con 0x7f02480a96b0 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.013+0000 7f024cbaf640 10 monclient(hunting): authenticate will time out at 5300.336490s 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.014+0000 7f0246d9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f0248142a20 0x7f02480ac550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:36516/0 (socket says 172.21.15.159:36516) 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.014+0000 7f0246d9d640 1 -- 172.21.15.159:0/1733947187 learned_addr learned my addr 172.21.15.159:0/1733947187 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.015+0000 7f023effd640 10 client.?.objecter ms_handle_connect 0x7f0248142a20 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.015+0000 7f023effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.015+0000 7f023effd640 10 client.?.objecter ms_handle_connect 0x7f02480b4700 2024-03-25T17:29:25.018 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.015+0000 7f023effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552537950 0 0) 0x7f02480ae2b0 con 0x7f02480b4700 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): my global_id is 4114 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0228004eb0 con 0x7f02480b4700 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307394232 0 0) 0x7f02480ae060 con 0x7f0248142a20 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 10 monclient(hunting): my global_id is 4119 2024-03-25T17:29:25.019 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.017+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02480ae2b0 con 0x7f0248142a20 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3624175521 0 0) 0x7f0228004eb0 con 0x7f02480b4700 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02480ae060 con 0x7f02480b4700 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f02300018f0 con 0x7f02480b4700 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 10 monclient(hunting): dump: 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.023 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2914005172 0 0) 0x7f02480ae2b0 con 0x7f0248142a20 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0228004eb0 con 0x7f0248142a20 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0240001f30 con 0x7f0248142a20 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.022+0000 7f023effd640 10 monclient(hunting): dump: 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.024 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.025 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.025 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.025 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.025 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.025 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.025 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 895735853 0 0) 0x7f02480ae060 con 0x7f02480b4700 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 >> v1:172.21.15.134:6790/0 conn(0x7f02480a96b0 legacy=0x7f02481d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 >> v1:172.21.15.134:6789/0 conn(0x7f0248142a20 legacy=0x7f02480ac550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 1 monclient: found mon.b 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02481d8150 con 0x7f02480b4700 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f023effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.027 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 5 monclient: authenticate success, global_id 4114 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 10 monclient: _renew_subs 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02481d3fd0 con 0x7f02480b4700 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 10 monclient: _renew_subs 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02481d4f00 con 0x7f02480b4700 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.025+0000 7f024cbaf640 1 librados: init done 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.026+0000 7f024cbaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.026+0000 7f024cbaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.026+0000 7f024cbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.028 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.026+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02481d76b0 con 0x7f02480b4700 2024-03-25T17:29:25.054 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.054+0000 7f87c240f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:29:25.064 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.064+0000 7fed5a02b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T17:29:25.098 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.098+0000 7f87c240f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:29:25.108 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.108+0000 7fed5a02b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T17:29:25.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0230001d40 con 0x7f02480b4700 2024-03-25T17:29:25.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:25.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0230004a00 con 0x7f02480b4700 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 10 monclient: dump: 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0230004fb0 con 0x7f02480b4700 2024-03-25T17:29:25.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f024559a640 4 set_mon_vals no callback set 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3655118924 0 0) 0x7f0230006570 con 0x7f02480b4700 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 10 client.4114.objecter ms_dispatch 0x7f0248000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0230006970 con 0x7f02480b4700 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.126+0000 7f023effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:25.199 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.198+0000 7f024cbaf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "bee90ced-7048-4b40-a256-52b8e0df28b8", "id": 0}] 2024-03-25T17:29:25.199 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.198+0000 7f024cbaf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "bee90ced-7048-4b40-a256-52b8e0df28b8", "id": 0}] 2024-03-25T17:29:25.199 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.198+0000 7f024cbaf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.199 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.198+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd new", "uuid": "bee90ced-7048-4b40-a256-52b8e0df28b8", "id": 0} v 0) v1 -- 0x7f024804f730 con 0x7f02480b4700 2024-03-25T17:29:25.212 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.212+0000 7f87c240f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:29:25.229 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.229+0000 7fed5a02b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T17:29:25.284 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.284+0000 7f87c240f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:25.300 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.300+0000 7fed5a02b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:25.334 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.334+0000 7f87c240f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:29:25.348 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.348+0000 7fed5a02b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T17:29:25.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.369+0000 7f023effd640 1 -- 172.21.15.159:0/1733947187 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bee90ced-7048-4b40-a256-52b8e0df28b8", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3808123841 0 732707515) 0x7f0230006b10 con 0x7f02480b4700 2024-03-25T17:29:25.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.369+0000 7f023effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "bee90ced-7048-4b40-a256-52b8e0df28b8", "id": 0}] 2024-03-25T17:29:25.370 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.369+0000 7f023effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:25.370 INFO:teuthology.orchestra.run.smithi159.stdout:0 2024-03-25T17:29:25.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 10 librados: watch_flush enter 2024-03-25T17:29:25.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 10 librados: watch_flush exit 2024-03-25T17:29:25.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-25T17:29:25.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 10 client.4114.objecter successfully canceled tick 2024-03-25T17:29:25.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 10 monclient: shutdown 2024-03-25T17:29:25.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:25.371 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 >> v1:172.21.15.159:6789/0 conn(0x7f02480b4700 legacy=0x7f02481d4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 shutdown_connections 2024-03-25T17:29:25.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 >> 172.21.15.159:0/1733947187 conn(0x7f02480668e0 msgr2=0x7f024813c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:25.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 shutdown_connections 2024-03-25T17:29:25.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 1 -- 172.21.15.159:0/1733947187 wait complete. 2024-03-25T17:29:25.372 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.370+0000 7f024cbaf640 1 librados: shutdown 2024-03-25T17:29:25.381 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 78238efc-fb2f-47d4-a7f8-25baa1cbb58b 1 2024-03-25T17:29:25.388 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.388+0000 7f87c240f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:29:25.405 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.404+0000 7fed5a02b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T17:29:25.435 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.434+0000 7f87c240f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:29:25.454 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.454+0000 7fed5a02b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.480+0000 7f2150764640 10 monclient: get_monmap_and_config 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.480+0000 7f2150764640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.480+0000 7f2150764640 10 monclient: monmap: 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:25.481326+0000 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:25.481326+0000 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:25.481 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:25.482 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:25.482 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.482 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.480+0000 7f2150764640 1 Processor -- start 2024-03-25T17:29:25.482 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.480+0000 7f2150764640 1 -- start start 2024-03-25T17:29:25.482 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.480+0000 7f2150764640 10 monclient: init 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 10 monclient(hunting): picked mon.noname-a con 0x7f21480b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 10 monclient(hunting): picked mon.noname-c con 0x7f2148142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 10 monclient(hunting): picked mon.noname-b con 0x7f21480a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480a9d60 con 0x7f21480b4700 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214813ff10 con 0x7f2148142a20 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21480aa1f0 con 0x7f21480a96b0 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.481+0000 7f2150764640 10 monclient(hunting): authenticate will time out at 5300.804268s 2024-03-25T17:29:25.483 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.482+0000 7f214dcff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f2148142a20 0x7f214813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35470/0 (socket says 172.21.15.159:35470) 2024-03-25T17:29:25.484 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.482+0000 7f214dcff640 1 -- 172.21.15.159:0/4152795862 learned_addr learned my addr 172.21.15.159:0/4152795862 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15644807 0 0) 0x7f214813ff10 con 0x7f2148142a20 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): my global_id is 4117 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2134003520 con 0x7f2148142a20 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347733801 0 0) 0x7f21480aa1f0 con 0x7f21480a96b0 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.485 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 10 monclient(hunting): my global_id is 34106 2024-03-25T17:29:25.486 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.484+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214813ff10 con 0x7f21480a96b0 2024-03-25T17:29:25.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.486+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641410967 0 0) 0x7f21480a9d60 con 0x7f21480b4700 2024-03-25T17:29:25.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.486+0000 7f214d4fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.486+0000 7f214d4fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.486+0000 7f214d4fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.486+0000 7f214d4fe640 10 monclient(hunting): my global_id is 4122 2024-03-25T17:29:25.487 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.486+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21480aa1f0 con 0x7f21480b4700 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4000276859 0 0) 0x7f2134003520 con 0x7f2148142a20 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 >> v1:172.21.15.134:6790/0 conn(0x7f21480a96b0 legacy=0x7f214813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 >> v1:172.21.15.134:6789/0 conn(0x7f21480b4700 legacy=0x7f2148142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 1 monclient: found mon.noname-c 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21480b4ae0 con 0x7f2148142a20 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.489 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2138001690 con 0x7f2148142a20 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f2150764640 5 monclient: authenticate success, global_id 4117 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f2150764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f214d4fe640 10 monclient: dump: 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.490 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.491 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.491 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.491 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.488+0000 7f2150764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:25.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21380017f0 con 0x7f2148142a20 2024-03-25T17:29:25.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f214d4fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:25.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f214d4fe640 1 -- 172.21.15.159:0/4152795862 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2138001c00 con 0x7f2148142a20 2024-03-25T17:29:25.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f214d4fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f214d4fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:25.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f214d4fe640 10 monclient: dump: 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f2150764640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f2150764640 4 set_mon_vals no callback set 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f2150764640 10 monclient: shutdown 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f2150764640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.490+0000 7f2150764640 1 -- 172.21.15.159:0/4152795862 >> v1:172.21.15.159:6789/0 conn(0x7f2148142a20 legacy=0x7f214813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 1 -- 172.21.15.159:0/4152795862 shutdown_connections 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 1 -- 172.21.15.159:0/4152795862 >> 172.21.15.159:0/4152795862 conn(0x7f21480668e0 msgr2=0x7f2148066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 1 -- 172.21.15.159:0/4152795862 shutdown_connections 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 1 -- 172.21.15.159:0/4152795862 wait complete. 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 10 monclient: monmap: 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:25.494 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 1 librados: starting msgr at 2024-03-25T17:29:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.491+0000 7f2150764640 1 librados: starting objecter 2024-03-25T17:29:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 1 Processor -- start 2024-03-25T17:29:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 1 -- start start 2024-03-25T17:29:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 1 librados: setting wanted keys 2024-03-25T17:29:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 1 librados: calling monclient init 2024-03-25T17:29:25.495 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient: init 2024-03-25T17:29:25.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:25.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient(hunting): picked mon.noname-b con 0x7f21480a96b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:25.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient(hunting): picked mon.noname-a con 0x7f21480b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:25.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient(hunting): picked mon.noname-c con 0x7f2148142a20 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:25.497 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214814eca0 con 0x7f21480b4700 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21481d74d0 con 0x7f21480a96b0 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21481d8350 con 0x7f2148142a20 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.492+0000 7f2150764640 10 monclient(hunting): authenticate will time out at 5300.815257s 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f214e500640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f21480a96b0 0x7f21481d7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35482/0 (socket says 172.21.15.159:35482) 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f214e500640 1 -- 172.21.15.159:0/2254631874 learned_addr learned my addr 172.21.15.159:0/2254631874 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f213effd640 10 client.?.objecter ms_handle_connect 0x7f21480a96b0 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f213effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f213effd640 10 client.?.objecter ms_handle_connect 0x7f2148142a20 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f213effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f213effd640 10 client.?.objecter ms_handle_connect 0x7f21480b4700 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.493+0000 7f213effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742391186 0 0) 0x7f21480b4ae0 con 0x7f21480b4700 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.498 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): my global_id is 4125 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c004eb0 con 0x7f21480b4700 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421538324 0 0) 0x7f21481d74d0 con 0x7f21480a96b0 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 10 monclient(hunting): my global_id is 4120 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.495+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21480b4ae0 con 0x7f21480a96b0 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.496+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506809594 0 0) 0x7f21481d8350 con 0x7f2148142a20 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.496+0000 7f213effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.496+0000 7f213effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.496+0000 7f213effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.496+0000 7f213effd640 10 monclient(hunting): my global_id is 34109 2024-03-25T17:29:25.499 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.496+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21481d74d0 con 0x7f2148142a20 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1639184510 0 0) 0x7f212c004eb0 con 0x7f21480b4700 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21481d8350 con 0x7f21480b4700 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2138001960 con 0x7f21480b4700 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 10 monclient(hunting): dump: 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.501 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309141985 0 0) 0x7f21480b4ae0 con 0x7f21480a96b0 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f212c004eb0 con 0x7f21480a96b0 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f21440019f0 con 0x7f21480a96b0 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.500+0000 7f213effd640 10 monclient(hunting): dump: 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.502 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.503 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.503 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.503 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.503 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.503 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.503 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.504 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2488918746 0 0) 0x7f21481d8350 con 0x7f21480b4700 2024-03-25T17:29:25.504 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 >> v1:172.21.15.134:6790/0 conn(0x7f2148142a20 legacy=0x7f21481d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.504 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 >> v1:172.21.15.159:6789/0 conn(0x7f21480a96b0 legacy=0x7f21481d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.504 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 1 monclient: found mon.a 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21480abd90 con 0x7f21480b4700 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f213effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 5 monclient: authenticate success, global_id 4125 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 10 monclient: _renew_subs 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21481d7720 con 0x7f21480b4700 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 10 monclient: _renew_subs 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:25.505 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21480ad240 con 0x7f21480b4700 2024-03-25T17:29:25.506 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.503+0000 7f2150764640 1 librados: init done 2024-03-25T17:29:25.506 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.505+0000 7f2150764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.506 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.505+0000 7f2150764640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.506 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.505+0000 7f2150764640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:25.506 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.505+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214804f730 con 0x7f21480b4700 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2138001b80 con 0x7f21480b4700 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2138004c30 con 0x7f21480b4700 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 10 monclient: dump: 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.509 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2138005120 con 0x7f21480b4700 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f214ccfd640 4 set_mon_vals no callback set 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3938062114 0 0) 0x7f2138006450 con 0x7f21480b4700 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 10 client.4125.objecter ms_dispatch 0x7f2148000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T17:29:25.510 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.508+0000 7f213effd640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T17:29:25.618 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.618+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2138006830 con 0x7f21480b4700 2024-03-25T17:29:25.618 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.618+0000 7f213effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.619 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.618+0000 7f213effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.690+0000 7f2150764640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "78238efc-fb2f-47d4-a7f8-25baa1cbb58b", "id": 1}] 2024-03-25T17:29:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.691+0000 7f2150764640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "78238efc-fb2f-47d4-a7f8-25baa1cbb58b", "id": 1}] 2024-03-25T17:29:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.691+0000 7f2150764640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:25.691 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.691+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd new", "uuid": "78238efc-fb2f-47d4-a7f8-25baa1cbb58b", "id": 1} v 0) v1 -- 0x7f21481d78b0 con 0x7f21480b4700 2024-03-25T17:29:25.772 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.772+0000 7f213effd640 1 -- 172.21.15.159:0/2254631874 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "78238efc-fb2f-47d4-a7f8-25baa1cbb58b", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3040185931 0 940483276) 0x7f2138006a40 con 0x7f21480b4700 2024-03-25T17:29:25.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.772+0000 7f213effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "78238efc-fb2f-47d4-a7f8-25baa1cbb58b", "id": 1}] 2024-03-25T17:29:25.773 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.772+0000 7f213effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:25.773 INFO:teuthology.orchestra.run.smithi159.stdout:1 2024-03-25T17:29:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 10 librados: watch_flush enter 2024-03-25T17:29:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 10 librados: watch_flush exit 2024-03-25T17:29:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T17:29:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 10 client.4125.objecter successfully canceled tick 2024-03-25T17:29:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 10 monclient: shutdown 2024-03-25T17:29:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:25.774 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 >> v1:172.21.15.134:6789/0 conn(0x7f21480b4700 legacy=0x7f21480acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 shutdown_connections 2024-03-25T17:29:25.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 >> 172.21.15.159:0/2254631874 conn(0x7f21480668e0 msgr2=0x7f21480b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:25.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 shutdown_connections 2024-03-25T17:29:25.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 1 -- 172.21.15.159:0/2254631874 wait complete. 2024-03-25T17:29:25.775 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.773+0000 7f2150764640 1 librados: shutdown 2024-03-25T17:29:25.784 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 3e547d19-49b0-4ad4-9b35-3c9702151be7 2 2024-03-25T17:29:25.839 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.838+0000 7f87c240f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:29:25.856 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.855+0000 7fed5a02b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.924+0000 7f01fa7d6640 10 monclient: get_monmap_and_config 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.924+0000 7f01fa7d6640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.924+0000 7f01fa7d6640 10 monclient: monmap: 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:25.925455+0000 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:25.925455+0000 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:25.925 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:25.926 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:25.926 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 1 Processor -- start 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 1 -- start start 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient: init 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f01f40b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f01f4142a20 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f01f40a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f40a9d60 con 0x7f01f40a96b0 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f413ff10 con 0x7f01f40b4700 2024-03-25T17:29:25.927 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f40aa1f0 con 0x7f01f4142a20 2024-03-25T17:29:25.928 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.928 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.925+0000 7f01fa7d6640 10 monclient(hunting): authenticate will time out at 5301.248384s 2024-03-25T17:29:25.928 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.926+0000 7f01f3fff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f01f40b4700 0x7f01f4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35490/0 (socket says 172.21.15.159:35490) 2024-03-25T17:29:25.928 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.926+0000 7f01f3fff640 1 -- 172.21.15.159:0/2290288443 learned_addr learned my addr 172.21.15.159:0/2290288443 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402863419 0 0) 0x7f01f40aa1f0 con 0x7f01f4142a20 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): my global_id is 34112 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01dc003520 con 0x7f01f4142a20 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406906501 0 0) 0x7f01f413ff10 con 0x7f01f40b4700 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.929 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T17:29:25.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.928+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f40aa1f0 con 0x7f01f40b4700 2024-03-25T17:29:25.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.929+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171444305 0 0) 0x7f01f40a9d60 con 0x7f01f40a96b0 2024-03-25T17:29:25.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.929+0000 7f01f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.929+0000 7f01f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.929+0000 7f01f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.929+0000 7f01f2ffd640 10 monclient(hunting): my global_id is 4128 2024-03-25T17:29:25.930 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.929+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f413ff10 con 0x7f01f40a96b0 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2322567154 0 0) 0x7f01f40aa1f0 con 0x7f01f40b4700 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 >> v1:172.21.15.134:6790/0 conn(0x7f01f4142a20 legacy=0x7f01f413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 >> v1:172.21.15.134:6789/0 conn(0x7f01f40a96b0 legacy=0x7f01f413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 1 monclient: found mon.noname-c 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f40b4ae0 con 0x7f01f40b4700 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.933 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01fa7d6640 5 monclient: authenticate success, global_id 4123 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01fa7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f01e40015f0 con 0x7f01f40b4700 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01f2ffd640 10 monclient: dump: 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.934 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.935 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.935 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.932+0000 7f01fa7d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.933+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01e4001750 con 0x7f01f40b4700 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01f2ffd640 1 -- 172.21.15.159:0/2290288443 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f01e4001b60 con 0x7f01f40b4700 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 4 set_mon_vals no callback set 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:25.936 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01f2ffd640 10 monclient: dump: 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 10 monclient: shutdown 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 1 -- 172.21.15.159:0/2290288443 >> v1:172.21.15.159:6789/0 conn(0x7f01f40b4700 legacy=0x7f01f4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.937 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 1 -- 172.21.15.159:0/2290288443 shutdown_connections 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 1 -- 172.21.15.159:0/2290288443 >> 172.21.15.159:0/2290288443 conn(0x7f01f40668e0 msgr2=0x7f01f4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 1 -- 172.21.15.159:0/2290288443 shutdown_connections 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.934+0000 7f01fa7d6640 1 -- 172.21.15.159:0/2290288443 wait complete. 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.935+0000 7f01fa7d6640 10 monclient: build_initial_monmap 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.935+0000 7f01fa7d6640 10 monclient: monmap: 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:25.938 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.935+0000 7f01fa7d6640 1 librados: starting msgr at 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.935+0000 7f01fa7d6640 1 librados: starting objecter 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.935+0000 7f01fa7d6640 1 Processor -- start 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 1 -- start start 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 1 librados: setting wanted keys 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 1 librados: calling monclient init 2024-03-25T17:29:25.939 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient: init 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient(hunting): picked mon.noname-a con 0x7f01f40a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient(hunting): picked mon.noname-c con 0x7f01f40b4700 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient(hunting): picked mon.noname-b con 0x7f01f4142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f41d4fe0 con 0x7f01f40a96b0 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f41d7590 con 0x7f01f4142a20 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01f413cca0 con 0x7f01f40b4700 2024-03-25T17:29:25.941 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.936+0000 7f01fa7d6640 10 monclient(hunting): authenticate will time out at 5301.259179s 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f8d73640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f01f4142a20 0x7f01f41e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35494/0 (socket says 172.21.15.159:35494) 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f8d73640 1 -- 172.21.15.159:0/982527402 learned_addr learned my addr 172.21.15.159:0/982527402 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f0ff9640 10 client.?.objecter ms_handle_connect 0x7f01f40b4700 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f0ff9640 10 client.?.objecter ms_handle_connect 0x7f01f40a96b0 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f0ff9640 10 client.?.objecter ms_handle_connect 0x7f01f4142a20 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.937+0000 7f01f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092506455 0 0) 0x7f01f41d4fe0 con 0x7f01f40a96b0 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.942 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): my global_id is 4131 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e0004eb0 con 0x7f01f40a96b0 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698232779 0 0) 0x7f01f41d7590 con 0x7f01f4142a20 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 10 monclient(hunting): my global_id is 4126 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.939+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f41d4fe0 con 0x7f01f4142a20 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.940+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008308743 0 0) 0x7f01f413cca0 con 0x7f01f40b4700 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.940+0000 7f01f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.940+0000 7f01f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.940+0000 7f01f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.940+0000 7f01f0ff9640 10 monclient(hunting): my global_id is 34115 2024-03-25T17:29:25.943 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.940+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f41d7590 con 0x7f01f40b4700 2024-03-25T17:29:25.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151499164 0 0) 0x7f01f41d4fe0 con 0x7f01f4142a20 2024-03-25T17:29:25.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f413cca0 con 0x7f01f4142a20 2024-03-25T17:29:25.944 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f01ec001cf0 con 0x7f01f4142a20 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.945 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:25.946 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2910115266 0 0) 0x7f01e0004eb0 con 0x7f01f40a96b0 2024-03-25T17:29:25.946 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01f41d4fe0 con 0x7f01f40a96b0 2024-03-25T17:29:25.946 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f01e4001a60 con 0x7f01f40a96b0 2024-03-25T17:29:25.946 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.946 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.944+0000 7f01f0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.945+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1895354204 0 0) 0x7f01f41d7590 con 0x7f01f40b4700 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.945+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01e0004eb0 con 0x7f01f40b4700 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.945+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f01e8001850 con 0x7f01f40b4700 2024-03-25T17:29:25.947 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.945+0000 7f01f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.945+0000 7f01f0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.945+0000 7f01f0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:25.948 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:25.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 913346811 0 0) 0x7f01f413cca0 con 0x7f01f4142a20 2024-03-25T17:29:25.949 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 >> v1:172.21.15.134:6790/0 conn(0x7f01f40b4700 legacy=0x7f01f413b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 >> v1:172.21.15.134:6789/0 conn(0x7f01f40a96b0 legacy=0x7f01f41d7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 1 monclient: found mon.b 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f41d81c0 con 0x7f01f4142a20 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 10 monclient: _finish_auth 0 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01fa7d6640 5 monclient: authenticate success, global_id 4126 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.946+0000 7f01fa7d6640 10 monclient: _renew_subs 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.947+0000 7f01fa7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.947+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f413baa0 con 0x7f01f4142a20 2024-03-25T17:29:25.950 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.947+0000 7f01fa7d6640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.947+0000 7f01fa7d6640 10 monclient: _renew_subs 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.947+0000 7f01fa7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.947+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f413bf80 con 0x7f01f4142a20 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.947+0000 7f01fa7d6640 1 librados: init done 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.948+0000 7f01fa7d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.948+0000 7f01fa7d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.948+0000 7f01fa7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:25.951 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:25.948+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01f404f730 con 0x7f01f4142a20 2024-03-25T17:29:25.980 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:25.979+0000 7f87c240f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:29:25.998 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:25.997+0000 7fed5a02b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T17:29:26.048 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01ec003990 con 0x7f01f4142a20 2024-03-25T17:29:26.048 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f01ec003d20 con 0x7f01f4142a20 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f0ff9640 10 monclient: dump: 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.049 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f01ec0042d0 con 0x7f01f4142a20 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 4094918456 0 0) 0x7f01ec006920 con 0x7f01f4142a20 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 10 client.4126.objecter ms_dispatch 0x7f01f4000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.047+0000 7f01f27fc640 4 set_mon_vals no callback set 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f01ec018de0 con 0x7f01f4142a20 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:26.050 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.048+0000 7f01f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:26.105 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:26.104+0000 7f87c240f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:29:26.115 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:26.114+0000 7fed5a02b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T17:29:26.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.120+0000 7f01fa7d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3e547d19-49b0-4ad4-9b35-3c9702151be7", "id": 2}] 2024-03-25T17:29:26.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.120+0000 7f01fa7d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3e547d19-49b0-4ad4-9b35-3c9702151be7", "id": 2}] 2024-03-25T17:29:26.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.120+0000 7f01fa7d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:26.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.120+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3e547d19-49b0-4ad4-9b35-3c9702151be7", "id": 2} v 0) v1 -- 0x7f01f40b7330 con 0x7f01f4142a20 2024-03-25T17:29:26.151 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:26.151+0000 7f87c240f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:29:26.163 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:26.162+0000 7fed5a02b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T17:29:26.180 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.179+0000 7f01f0ff9640 1 -- 172.21.15.159:0/982527402 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3e547d19-49b0-4ad4-9b35-3c9702151be7", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2354266566 0 216599125) 0x7f01ec006500 con 0x7f01f4142a20 2024-03-25T17:29:26.180 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.179+0000 7f01f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3e547d19-49b0-4ad4-9b35-3c9702151be7", "id": 2}] 2024-03-25T17:29:26.180 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.179+0000 7f01f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:26.180 INFO:teuthology.orchestra.run.smithi159.stdout:2 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 10 librados: watch_flush enter 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 10 librados: watch_flush exit 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 10 client.4126.objecter successfully canceled tick 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 10 monclient: shutdown 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 >> v1:172.21.15.159:6789/0 conn(0x7f01f4142a20 legacy=0x7f01f41e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 shutdown_connections 2024-03-25T17:29:26.181 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 >> 172.21.15.159:0/982527402 conn(0x7f01f40668e0 msgr2=0x7f01f40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:26.182 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 shutdown_connections 2024-03-25T17:29:26.182 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 1 -- 172.21.15.159:0/982527402 wait complete. 2024-03-25T17:29:26.182 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.180+0000 7f01fa7d6640 1 librados: shutdown 2024-03-25T17:29:26.191 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new dc538434-60ca-4c73-9226-d24a8316d9a4 3 2024-03-25T17:29:26.267 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:26.266+0000 7f87c240f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:26.282 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:26.281+0000 7fed5a02b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.330+0000 7f0022d8f640 10 monclient: get_monmap_and_config 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.330+0000 7f0022d8f640 10 monclient: build_initial_monmap 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.330+0000 7f0022d8f640 10 monclient: monmap: 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:26.331749+0000 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:26.331749+0000 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:26.331 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.332 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:26.332 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:26.332 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:26.332 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.331+0000 7f0022d8f640 1 Processor -- start 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.331+0000 7f0022d8f640 1 -- start start 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.331+0000 7f0022d8f640 10 monclient: init 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f001c0b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f001c142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f001c0a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c0a9d60 con 0x7f001c0b4700 2024-03-25T17:29:26.333 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c13ff10 con 0x7f001c142a20 2024-03-25T17:29:26.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c0aa1f0 con 0x7f001c0a96b0 2024-03-25T17:29:26.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:26.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f0022d8f640 10 monclient(hunting): authenticate will time out at 5301.654722s 2024-03-25T17:29:26.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f002132c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f001c0a96b0 0x7f001c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:32842/0 (socket says 172.21.15.159:32842) 2024-03-25T17:29:26.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f002132c640 1 -- 172.21.15.159:0/1551028739 learned_addr learned my addr 172.21.15.159:0/1551028739 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:26.334 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.332+0000 7f001bfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f001c142a20 0x7f001c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35504/0 (socket says 172.21.15.159:35504) 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379970243 0 0) 0x7f001c13ff10 con 0x7f001c142a20 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): my global_id is 4129 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f000c003520 con 0x7f001c142a20 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062716132 0 0) 0x7f001c0a9d60 con 0x7f001c0b4700 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): my global_id is 4134 2024-03-25T17:29:26.336 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c13ff10 con 0x7f001c0b4700 2024-03-25T17:29:26.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112168923 0 0) 0x7f001c0aa1f0 con 0x7f001c0a96b0 2024-03-25T17:29:26.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 10 monclient(hunting): my global_id is 34118 2024-03-25T17:29:26.337 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.335+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c0a9d60 con 0x7f001c0a96b0 2024-03-25T17:29:26.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2709409765 0 0) 0x7f000c003520 con 0x7f001c142a20 2024-03-25T17:29:26.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 >> v1:172.21.15.134:6790/0 conn(0x7f001c0a96b0 legacy=0x7f001c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 >> v1:172.21.15.134:6789/0 conn(0x7f001c0b4700 legacy=0x7f001c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:26.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 1 monclient: found mon.noname-c 2024-03-25T17:29:26.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:26.339 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c0b4ae0 con 0x7f001c142a20 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0008001690 con 0x7f001c142a20 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f0022d8f640 5 monclient: authenticate success, global_id 4129 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f0022d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f001b7fe640 10 monclient: dump: 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.340 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.341 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.341 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.341 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.341 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.341 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.338+0000 7f0022d8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:26.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00080017f0 con 0x7f001c142a20 2024-03-25T17:29:26.342 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f001b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f001b7fe640 1 -- 172.21.15.159:0/1551028739 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0008001c40 con 0x7f001c142a20 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f001b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f001b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f001b7fe640 10 monclient: dump: 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f0022d8f640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:26.343 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f0022d8f640 4 set_mon_vals no callback set 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f0022d8f640 10 monclient: shutdown 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f0022d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f0022d8f640 1 -- 172.21.15.159:0/1551028739 >> v1:172.21.15.159:6789/0 conn(0x7f001c142a20 legacy=0x7f001c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f0022d8f640 1 -- 172.21.15.159:0/1551028739 shutdown_connections 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.340+0000 7f0022d8f640 1 -- 172.21.15.159:0/1551028739 >> 172.21.15.159:0/1551028739 conn(0x7f001c0668e0 msgr2=0x7f001c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.341+0000 7f0022d8f640 1 -- 172.21.15.159:0/1551028739 shutdown_connections 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.341+0000 7f0022d8f640 1 -- 172.21.15.159:0/1551028739 wait complete. 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.341+0000 7f0022d8f640 10 monclient: build_initial_monmap 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.341+0000 7f0022d8f640 10 monclient: monmap: 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:26.344 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.341+0000 7f0022d8f640 1 librados: starting msgr at 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.341+0000 7f0022d8f640 1 librados: starting objecter 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 1 Processor -- start 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 1 -- start start 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 1 librados: setting wanted keys 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 1 librados: calling monclient init 2024-03-25T17:29:26.345 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient: init 2024-03-25T17:29:26.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:26.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:26.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient(hunting): picked mon.noname-c con 0x7f001c0a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:26.347 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient(hunting): picked mon.noname-a con 0x7f001c0b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient(hunting): picked mon.noname-b con 0x7f001c142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c0ae580 con 0x7f001c0b4700 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c13aef0 con 0x7f001c142a20 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c14ea20 con 0x7f001c0a96b0 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.342+0000 7f0022d8f640 10 monclient(hunting): authenticate will time out at 5301.665452s 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f0020b2b640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f001c0a96b0 0x7f001c139b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:32854/0 (socket says 172.21.15.159:32854) 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f0020b2b640 1 -- 172.21.15.159:0/3290415616 learned_addr learned my addr 172.21.15.159:0/3290415616 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f00197fa640 10 client.?.objecter ms_handle_connect 0x7f001c0a96b0 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f00197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f00197fa640 10 client.?.objecter ms_handle_connect 0x7f001c0b4700 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f00197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f00197fa640 10 client.?.objecter ms_handle_connect 0x7f001c142a20 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.343+0000 7f00197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:26.348 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255527584 0 0) 0x7f001c0ae580 con 0x7f001c0b4700 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): my global_id is 4137 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0000004eb0 con 0x7f001c0b4700 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238868184 0 0) 0x7f001c14ea20 con 0x7f001c0a96b0 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 10 monclient(hunting): my global_id is 34121 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.345+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c0ae580 con 0x7f001c0a96b0 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.346+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 448327760 0 0) 0x7f001c13aef0 con 0x7f001c142a20 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.346+0000 7f00197fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.346+0000 7f00197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.346+0000 7f00197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.349 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.346+0000 7f00197fa640 10 monclient(hunting): my global_id is 4132 2024-03-25T17:29:26.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.346+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c14ea20 con 0x7f001c142a20 2024-03-25T17:29:26.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.349+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4022119207 0 0) 0x7f001c0ae580 con 0x7f001c0a96b0 2024-03-25T17:29:26.350 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.349+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f001c13aef0 con 0x7f001c0a96b0 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.349+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0010001a50 con 0x7f001c0a96b0 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.349+0000 7f00197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.349+0000 7f00197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.349+0000 7f00197fa640 10 monclient(hunting): dump: 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.351 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.349+0000 7f00197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.350+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224722881 0 0) 0x7f0000004eb0 con 0x7f001c0b4700 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.350+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f001c0ae580 con 0x7f001c0b4700 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.350+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0008001b00 con 0x7f001c0b4700 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.350+0000 7f00197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.350+0000 7f00197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.350+0000 7f00197fa640 10 monclient(hunting): dump: 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.352 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.351+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4125694359 0 0) 0x7f001c14ea20 con 0x7f001c142a20 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.351+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0000004eb0 con 0x7f001c142a20 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.351+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0014001bd0 con 0x7f001c142a20 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.351+0000 7f00197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.351+0000 7f00197fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.351+0000 7f00197fa640 10 monclient(hunting): dump: 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.353 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.354 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.354 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.354 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.354 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.354 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.354 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.354 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1472740231 0 0) 0x7f001c0ae580 con 0x7f001c0b4700 2024-03-25T17:29:26.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 >> v1:172.21.15.134:6790/0 conn(0x7f001c0a96b0 legacy=0x7f001c139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.355 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 >> v1:172.21.15.159:6789/0 conn(0x7f001c142a20 legacy=0x7f001c1e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 10 monclient: _finish_hunting 0 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 1 monclient: found mon.a 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c14ebb0 con 0x7f001c0b4700 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 10 monclient: _finish_auth 0 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 10 monclient: _check_auth_tickets 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f00197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 5 monclient: authenticate success, global_id 4137 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 10 monclient: _renew_subs 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c13a990 con 0x7f001c0b4700 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 10 monclient: _renew_subs 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.356 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c0ad110 con 0x7f001c0b4700 2024-03-25T17:29:26.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.354+0000 7f0022d8f640 1 librados: init done 2024-03-25T17:29:26.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.355+0000 7f0022d8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:26.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.355+0000 7f0022d8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:26.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.355+0000 7f0022d8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.357 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.355+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001c0ad410 con 0x7f001c0b4700 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0008001220 con 0x7f001c0b4700 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0008001d80 con 0x7f001c0b4700 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 10 monclient: dump: 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.359 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0008005120 con 0x7f001c0b4700 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f001affd640 4 set_mon_vals no callback set 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2393773486 0 0) 0x7f00080067d0 con 0x7f001c0b4700 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 10 client.4137.objecter ms_dispatch 0x7f001c000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T17:29:26.360 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.358+0000 7f00197fa640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T17:29:26.366 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:26.366+0000 7f87c240f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:29:26.385 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:26.385+0000 7fed5a02b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T17:29:26.452 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.451+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f00080063f0 con 0x7f001c0b4700 2024-03-25T17:29:26.452 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.451+0000 7f00197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:26.452 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.451+0000 7f00197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:26.523 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:26.523+0000 7f87c240f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:26.524 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.523+0000 7f0022d8f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dc538434-60ca-4c73-9226-d24a8316d9a4", "id": 3}] 2024-03-25T17:29:26.524 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.523+0000 7f0022d8f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dc538434-60ca-4c73-9226-d24a8316d9a4", "id": 3}] 2024-03-25T17:29:26.524 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.523+0000 7f0022d8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.524 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.523+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dc538434-60ca-4c73-9226-d24a8316d9a4", "id": 3} v 0) v1 -- 0x7f001c0ae3c0 con 0x7f001c0b4700 2024-03-25T17:29:26.545 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:26.544+0000 7fed5a02b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T17:29:26.569 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:26.569+0000 7f87c240f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:26.583 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.582+0000 7f00197fa640 1 -- 172.21.15.159:0/3290415616 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc538434-60ca-4c73-9226-d24a8316d9a4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 4038452641 0 525046306) 0x7f00080069e0 con 0x7f001c0b4700 2024-03-25T17:29:26.583 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.582+0000 7f00197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dc538434-60ca-4c73-9226-d24a8316d9a4", "id": 3}] 2024-03-25T17:29:26.583 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.582+0000 7f00197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:26.584 INFO:teuthology.orchestra.run.smithi159.stdout:3 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 10 librados: watch_flush enter 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 10 librados: watch_flush exit 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 10 client.4137.objecter successfully canceled tick 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 10 monclient: shutdown 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 >> v1:172.21.15.134:6789/0 conn(0x7f001c0b4700 legacy=0x7f001c13a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 shutdown_connections 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 >> 172.21.15.159:0/3290415616 conn(0x7f001c0668e0 msgr2=0x7f001c0b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 shutdown_connections 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 1 -- 172.21.15.159:0/3290415616 wait complete. 2024-03-25T17:29:26.585 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.583+0000 7f0022d8f640 1 librados: shutdown 2024-03-25T17:29:26.588 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:26.588+0000 7fed5a02b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T17:29:26.594 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 48552c57-30eb-4b13-b468-0417c4e03ad4 4 2024-03-25T17:29:26.694 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.693+0000 7fa9d98e6640 10 monclient: get_monmap_and_config 2024-03-25T17:29:26.694 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.693+0000 7fa9d98e6640 10 monclient: build_initial_monmap 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.693+0000 7fa9d98e6640 10 monclient: monmap: 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:26.694816+0000 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:26.694816+0000 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.694+0000 7fa9d98e6640 1 Processor -- start 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.694+0000 7fa9d98e6640 1 -- start start 2024-03-25T17:29:26.695 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.694+0000 7fa9d98e6640 10 monclient: init 2024-03-25T17:29:26.696 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.694+0000 7fa9d98e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.694+0000 7fa9d98e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d40b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d4142a20 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d40a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40a9d60 con 0x7fa9d4142a20 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d413ff10 con 0x7fa9d40b4700 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40aa1f0 con 0x7fa9d40a96b0 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d98e6640 10 monclient(hunting): authenticate will time out at 5302.017703s 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d2ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa9d40b4700 0x7fa9d4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35528/0 (socket says 172.21.15.159:35528) 2024-03-25T17:29:26.697 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.695+0000 7fa9d2ffd640 1 -- 172.21.15.159:0/2617754334 learned_addr learned my addr 172.21.15.159:0/2617754334 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:26.698 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.697+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889436833 0 0) 0x7fa9d40aa1f0 con 0x7fa9d40a96b0 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.697+0000 7fa9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.697+0000 7fa9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.697+0000 7fa9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.697+0000 7fa9d1ffb640 10 monclient(hunting): my global_id is 34124 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.697+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0003520 con 0x7fa9d40a96b0 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389412886 0 0) 0x7fa9d40a9d60 con 0x7fa9d4142a20 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): my global_id is 4140 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d40aa1f0 con 0x7fa9d4142a20 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493146491 0 0) 0x7fa9d413ff10 con 0x7fa9d40b4700 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.699 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 10 monclient(hunting): my global_id is 4135 2024-03-25T17:29:26.700 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.698+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d40a9d60 con 0x7fa9d40b4700 2024-03-25T17:29:26.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1637609107 0 0) 0x7fa9c0003520 con 0x7fa9d40a96b0 2024-03-25T17:29:26.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 >> v1:172.21.15.159:6789/0 conn(0x7fa9d40b4700 legacy=0x7fa9d4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 >> v1:172.21.15.134:6789/0 conn(0x7fa9d4142a20 legacy=0x7fa9d413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:29:26.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 1 monclient: found mon.noname-b 2024-03-25T17:29:26.702 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40b4ae0 con 0x7fa9d40a96b0 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa9c4001a10 con 0x7fa9d40a96b0 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d98e6640 5 monclient: authenticate success, global_id 34124 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d98e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d1ffb640 10 monclient: dump: 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.703 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.704 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.704 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.704 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.704 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.704 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.704 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.704 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.701+0000 7fa9d98e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:26.727 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:26.727+0000 7f87c240f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c4001b70 con 0x7fa9d40a96b0 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d1ffb640 1 -- 172.21.15.159:0/2617754334 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa9c4001f80 con 0x7fa9d40a96b0 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d1ffb640 10 monclient: dump: 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.728 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d98e6640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:26.729 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.727+0000 7fa9d98e6640 4 set_mon_vals no callback set 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.728+0000 7fa9d98e6640 10 monclient: shutdown 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.728+0000 7fa9d98e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.728+0000 7fa9d98e6640 1 -- 172.21.15.159:0/2617754334 >> v1:172.21.15.134:6790/0 conn(0x7fa9d40a96b0 legacy=0x7fa9d413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.728+0000 7fa9d98e6640 1 -- 172.21.15.159:0/2617754334 shutdown_connections 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.728+0000 7fa9d98e6640 1 -- 172.21.15.159:0/2617754334 >> 172.21.15.159:0/2617754334 conn(0x7fa9d40668e0 msgr2=0x7fa9d4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.728+0000 7fa9d98e6640 1 -- 172.21.15.159:0/2617754334 shutdown_connections 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.728+0000 7fa9d98e6640 1 -- 172.21.15.159:0/2617754334 wait complete. 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 10 monclient: build_initial_monmap 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 10 monclient: monmap: 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:26.732 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 1 librados: starting msgr at 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 1 librados: starting objecter 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 1 Processor -- start 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 1 -- start start 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 1 librados: setting wanted keys 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 1 librados: calling monclient init 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.729+0000 7fa9d98e6640 10 monclient: init 2024-03-25T17:29:26.733 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d40a96b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d40b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d4142a20 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40accd0 con 0x7fa9d40b4700 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40ae440 con 0x7fa9d40a96b0 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40ab880 con 0x7fa9d4142a20 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.730+0000 7fa9d98e6640 10 monclient(hunting): authenticate will time out at 5302.053161s 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.731+0000 7fa9d2ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fa9d40a96b0 0x7fa9d4139b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35544/0 (socket says 172.21.15.159:35544) 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.731+0000 7fa9d2ffd640 1 -- 172.21.15.159:0/3656517538 learned_addr learned my addr 172.21.15.159:0/3656517538 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:26.734 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.731+0000 7fa9cbfff640 10 client.?.objecter ms_handle_connect 0x7fa9d40b4700 2024-03-25T17:29:26.735 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.731+0000 7fa9cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:26.735 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.731+0000 7fa9cbfff640 10 client.?.objecter ms_handle_connect 0x7fa9d40a96b0 2024-03-25T17:29:26.735 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.731+0000 7fa9cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:26.735 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.733+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298693374 0 0) 0x7fa9d40accd0 con 0x7fa9d40b4700 2024-03-25T17:29:26.735 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.733+0000 7fa9cbfff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:26.735 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.733+0000 7fa9cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:26.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.733+0000 7fa9cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:26.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.733+0000 7fa9cbfff640 10 monclient(hunting): my global_id is 4143 2024-03-25T17:29:26.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.733+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9b4004eb0 con 0x7fa9d40b4700 2024-03-25T17:29:26.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.734+0000 7fa9cbfff640 10 client.?.objecter ms_handle_connect 0x7fa9d4142a20 2024-03-25T17:29:26.736 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.734+0000 7fa9cbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:26.739 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.738+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1308921708 0 0) 0x7fa9b4004eb0 con 0x7fa9d40b4700 2024-03-25T17:29:26.739 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.738+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d40accd0 con 0x7fa9d40b4700 2024-03-25T17:29:26.739 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.738+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa9bc001970 con 0x7fa9d40b4700 2024-03-25T17:29:26.739 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.738+0000 7fa9cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.739 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.738+0000 7fa9cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:26.739 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.738+0000 7fa9cbfff640 10 monclient(hunting): dump: 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.740 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.738+0000 7fa9cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1806651031 0 0) 0x7fa9d40accd0 con 0x7fa9d40b4700 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 >> v1:172.21.15.134:6790/0 conn(0x7fa9d4142a20 legacy=0x7fa9d41e2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 >> v1:172.21.15.159:6789/0 conn(0x7fa9d40a96b0 legacy=0x7fa9d4139b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 1 monclient: found mon.a 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40ad160 con 0x7fa9d40b4700 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 10 monclient: _finish_auth 0 2024-03-25T17:29:26.745 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 5 monclient: authenticate success, global_id 4143 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 10 monclient: _renew_subs 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d413a290 con 0x7fa9d40b4700 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 10 monclient: _renew_subs 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d40ac1c0 con 0x7fa9d40b4700 2024-03-25T17:29:26.746 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.744+0000 7fa9d98e6640 1 librados: init done 2024-03-25T17:29:26.747 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.745+0000 7fa9d98e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:26.747 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.745+0000 7fa9d98e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:26.747 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.745+0000 7fa9d98e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.747 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.745+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d40ace80 con 0x7fa9d40b4700 2024-03-25T17:29:26.756 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:26.755+0000 7fed5a02b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T17:29:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9bc003610 con 0x7fa9d40b4700 2024-03-25T17:29:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa9bc003780 con 0x7fa9d40b4700 2024-03-25T17:29:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:26.887 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9cbfff640 10 monclient: dump: 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa9bc003d30 con 0x7fa9d40b4700 2024-03-25T17:29:26.888 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.886+0000 7fa9d17fa640 4 set_mon_vals no callback set 2024-03-25T17:29:26.889 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2415176771 0 0) 0x7fa9bc005440 con 0x7fa9d40b4700 2024-03-25T17:29:26.890 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 10 client.4143.objecter ms_dispatch 0x7fa9d4000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T17:29:26.890 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T17:29:26.890 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T17:29:26.890 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T17:29:26.890 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa9bc004f30 con 0x7fa9d40b4700 2024-03-25T17:29:26.890 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:26.890 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.889+0000 7fa9cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:26.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.961+0000 7fa9d98e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "48552c57-30eb-4b13-b468-0417c4e03ad4", "id": 4}] 2024-03-25T17:29:26.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.961+0000 7fa9d98e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "48552c57-30eb-4b13-b468-0417c4e03ad4", "id": 4}] 2024-03-25T17:29:26.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.961+0000 7fa9d98e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:29:26.962 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.961+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd new", "uuid": "48552c57-30eb-4b13-b468-0417c4e03ad4", "id": 4} v 0) v1 -- 0x7fa9d413a5e0 con 0x7fa9d40b4700 2024-03-25T17:29:26.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9cbfff640 1 -- 172.21.15.159:0/3656517538 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48552c57-30eb-4b13-b468-0417c4e03ad4", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2447856788 0 1697012583) 0x7fa9bc0052a0 con 0x7fa9d40b4700 2024-03-25T17:29:26.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "48552c57-30eb-4b13-b468-0417c4e03ad4", "id": 4}] 2024-03-25T17:29:26.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:26.997 INFO:teuthology.orchestra.run.smithi159.stdout:4 2024-03-25T17:29:26.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9d98e6640 10 librados: watch_flush enter 2024-03-25T17:29:26.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9d98e6640 10 librados: watch_flush exit 2024-03-25T17:29:26.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9d98e6640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T17:29:26.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9d98e6640 10 client.4143.objecter successfully canceled tick 2024-03-25T17:29:26.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.996+0000 7fa9d98e6640 10 monclient: shutdown 2024-03-25T17:29:26.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.997+0000 7fa9d98e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:26.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.997+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 >> v1:172.21.15.134:6789/0 conn(0x7fa9d40b4700 legacy=0x7fa9d41df570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:26.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.997+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 shutdown_connections 2024-03-25T17:29:26.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.997+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 >> 172.21.15.159:0/3656517538 conn(0x7fa9d40668e0 msgr2=0x7fa9d40b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:26.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.997+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 shutdown_connections 2024-03-25T17:29:26.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.997+0000 7fa9d98e6640 1 -- 172.21.15.159:0/3656517538 wait complete. 2024-03-25T17:29:26.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:26.997+0000 7fa9d98e6640 1 librados: shutdown 2024-03-25T17:29:27.007 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 5f4c8c05-9231-4a4b-b76f-602253e33077 5 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.109+0000 7f7a66806640 10 monclient: get_monmap_and_config 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.109+0000 7f7a66806640 10 monclient: build_initial_monmap 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.109+0000 7f7a66806640 10 monclient: monmap: 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:27.110594+0000 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:27.110594+0000 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:27.110 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:27.111 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:27.111 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 1 Processor -- start 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 1 -- start start 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient: init 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient(hunting): picked mon.noname-c con 0x7f7a600b4700 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient(hunting): picked mon.noname-b con 0x7f7a60142a20 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:27.112 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient(hunting): picked mon.noname-a con 0x7f7a600a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:27.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a600a9d60 con 0x7f7a600a96b0 2024-03-25T17:29:27.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6013ff10 con 0x7f7a600b4700 2024-03-25T17:29:27.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a600aa1f0 con 0x7f7a60142a20 2024-03-25T17:29:27.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:27.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.110+0000 7f7a66806640 10 monclient(hunting): authenticate will time out at 5302.433545s 2024-03-25T17:29:27.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.111+0000 7f7a5f7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f7a60142a20 0x7f7a6013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:32870/0 (socket says 172.21.15.159:32870) 2024-03-25T17:29:27.113 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.111+0000 7f7a5f7fe640 1 -- 172.21.15.159:0/2985279702 learned_addr learned my addr 172.21.15.159:0/2985279702 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:27.114 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.113+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082587382 0 0) 0x7f7a600aa1f0 con 0x7f7a60142a20 2024-03-25T17:29:27.114 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.113+0000 7f7a5effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.114 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.113+0000 7f7a5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.114 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.113+0000 7f7a5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.114 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.113+0000 7f7a5effd640 10 monclient(hunting): my global_id is 34127 2024-03-25T17:29:27.114 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.113+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a48003520 con 0x7f7a60142a20 2024-03-25T17:29:27.115 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982274664 0 0) 0x7f7a6013ff10 con 0x7f7a600b4700 2024-03-25T17:29:27.115 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.115 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.115 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.115 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): my global_id is 4138 2024-03-25T17:29:27.115 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a600aa1f0 con 0x7f7a600b4700 2024-03-25T17:29:27.116 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2530956851 0 0) 0x7f7a600a9d60 con 0x7f7a600a96b0 2024-03-25T17:29:27.116 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.116 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.116 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.116 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 10 monclient(hunting): my global_id is 4146 2024-03-25T17:29:27.116 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.114+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6013ff10 con 0x7f7a600a96b0 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.116+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2497464713 0 0) 0x7f7a48003520 con 0x7f7a60142a20 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 >> v1:172.21.15.159:6789/0 conn(0x7f7a600b4700 legacy=0x7f7a60142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 >> v1:172.21.15.134:6789/0 conn(0x7f7a600a96b0 legacy=0x7f7a6013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 1 monclient: found mon.noname-b 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a600b4ae0 con 0x7f7a60142a20 2024-03-25T17:29:27.118 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 10 monclient: _finish_auth 0 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a66806640 5 monclient: authenticate success, global_id 34127 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a66806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7a54001650 con 0x7f7a60142a20 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a5effd640 10 monclient: dump: 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.119 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.120 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.120 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.120 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.120 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.120 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.117+0000 7f7a66806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a540018a0 con 0x7f7a60142a20 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a5effd640 1 -- 172.21.15.159:0/2985279702 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7a54001c30 con 0x7f7a60142a20 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a5effd640 10 monclient: dump: 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.121 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a66806640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.119+0000 7f7a66806640 4 set_mon_vals no callback set 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.120+0000 7f7a66806640 10 monclient: shutdown 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.120+0000 7f7a66806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.120+0000 7f7a66806640 1 -- 172.21.15.159:0/2985279702 >> v1:172.21.15.134:6790/0 conn(0x7f7a60142a20 legacy=0x7f7a6013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.120+0000 7f7a66806640 1 -- 172.21.15.159:0/2985279702 shutdown_connections 2024-03-25T17:29:27.122 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.120+0000 7f7a66806640 1 -- 172.21.15.159:0/2985279702 >> 172.21.15.159:0/2985279702 conn(0x7f7a600668e0 msgr2=0x7f7a60066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:27.123 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.120+0000 7f7a66806640 1 -- 172.21.15.159:0/2985279702 shutdown_connections 2024-03-25T17:29:27.123 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.120+0000 7f7a66806640 1 -- 172.21.15.159:0/2985279702 wait complete. 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 10 monclient: build_initial_monmap 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 10 monclient: monmap: 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.125 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 1 librados: starting msgr at 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 1 librados: starting objecter 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 1 Processor -- start 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 1 -- start start 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 1 librados: setting wanted keys 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 1 librados: calling monclient init 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.121+0000 7f7a66806640 10 monclient: init 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 10 monclient(hunting): picked mon.noname-b con 0x7f7a600a96b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 10 monclient(hunting): picked mon.noname-a con 0x7f7a600b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 10 monclient(hunting): picked mon.noname-c con 0x7f7a60142a20 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:27.126 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a6012ce30 con 0x7f7a600b4700 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a60133ce0 con 0x7f7a600a96b0 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a60133fc0 con 0x7f7a60142a20 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a66806640 10 monclient(hunting): authenticate will time out at 5302.444897s 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a64da3640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f7a60142a20 0x7f7a601e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.159:32882/0 (socket says 172.21.15.159:32882) 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a5ffff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7a600a96b0 0x7f7a601316d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:35564/0 (socket says 172.21.15.159:35564) 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.122+0000 7f7a64da3640 1 -- 172.21.15.159:0/1729125524 learned_addr learned my addr 172.21.15.159:0/1729125524 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.123+0000 7f7a5cff9640 10 client.?.objecter ms_handle_connect 0x7f7a60142a20 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.123+0000 7f7a5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.123+0000 7f7a5cff9640 10 client.?.objecter ms_handle_connect 0x7f7a600a96b0 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.123+0000 7f7a5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.123+0000 7f7a5cff9640 10 client.?.objecter ms_handle_connect 0x7f7a600b4700 2024-03-25T17:29:27.127 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.123+0000 7f7a5cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022233921 0 0) 0x7f7a60133fc0 con 0x7f7a60142a20 2024-03-25T17:29:27.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.128 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): my global_id is 34130 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a4c004eb0 con 0x7f7a60142a20 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623754758 0 0) 0x7f7a60133ce0 con 0x7f7a600a96b0 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): my global_id is 4141 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a60133fc0 con 0x7f7a600a96b0 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1422535884 0 0) 0x7f7a600b4ae0 con 0x7f7a600b4700 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.129 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.130 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 10 monclient(hunting): my global_id is 4149 2024-03-25T17:29:27.130 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.125+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a60133ce0 con 0x7f7a600b4700 2024-03-25T17:29:27.130 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.129+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 336234783 0 0) 0x7f7a4c004eb0 con 0x7f7a60142a20 2024-03-25T17:29:27.130 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.129+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a600b4ae0 con 0x7f7a60142a20 2024-03-25T17:29:27.130 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.129+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7a58001c70 con 0x7f7a60142a20 2024-03-25T17:29:27.130 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.129+0000 7f7a5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.130 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.129+0000 7f7a5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.129+0000 7f7a5cff9640 10 monclient(hunting): dump: 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.131 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.129+0000 7f7a5cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448924685 0 0) 0x7f7a60133fc0 con 0x7f7a600a96b0 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a4c004eb0 con 0x7f7a600a96b0 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7a50001930 con 0x7f7a600a96b0 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 10 monclient(hunting): dump: 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.133 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3261556181 0 0) 0x7f7a60133ce0 con 0x7f7a600b4700 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a60133fc0 con 0x7f7a600b4700 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7a540010e0 con 0x7f7a600b4700 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.132+0000 7f7a5cff9640 10 monclient(hunting): dump: 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.134 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.135 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.135 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2182291038 0 0) 0x7f7a600b4ae0 con 0x7f7a60142a20 2024-03-25T17:29:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 >> v1:172.21.15.159:6789/0 conn(0x7f7a600a96b0 legacy=0x7f7a601316d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 >> v1:172.21.15.134:6789/0 conn(0x7f7a600b4700 legacy=0x7f7a60131de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 10 monclient: _finish_hunting 0 2024-03-25T17:29:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 1 monclient: found mon.c 2024-03-25T17:29:27.136 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a601342a0 con 0x7f7a60142a20 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 10 monclient: _finish_auth 0 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 10 monclient: _check_auth_tickets 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 5 monclient: authenticate success, global_id 34130 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 10 monclient: _renew_subs 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a60134580 con 0x7f7a60142a20 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 10 client.34130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 10 monclient: _renew_subs 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:27.137 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a601311b0 con 0x7f7a60142a20 2024-03-25T17:29:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.133+0000 7f7a66806640 1 librados: init done 2024-03-25T17:29:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.134+0000 7f7a66806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.134+0000 7f7a66806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.134+0000 7f7a66806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:27.138 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.134+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a6004f730 con 0x7f7a60142a20 2024-03-25T17:29:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a58003990 con 0x7f7a60142a20 2024-03-25T17:29:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7a58003da0 con 0x7f7a60142a20 2024-03-25T17:29:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:27.322 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5cff9640 10 monclient: dump: 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7a58004350 con 0x7f7a60142a20 2024-03-25T17:29:27.323 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.321+0000 7f7a5e7fc640 4 set_mon_vals no callback set 2024-03-25T17:29:27.324 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 254366273 0 0) 0x7f7a58006ad0 con 0x7f7a60142a20 2024-03-25T17:29:27.325 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 10 client.34130.objecter ms_dispatch 0x7f7a60000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T17:29:27.325 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 3 client.34130.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T17:29:27.325 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 3 client.34130.objecter handle_osd_map decoding full epoch 7 2024-03-25T17:29:27.325 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 20 client.34130.objecter dump_active .. 0 homeless 2024-03-25T17:29:27.325 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7a58005b40 con 0x7f7a60142a20 2024-03-25T17:29:27.325 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:27.325 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.324+0000 7f7a5cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:27.397 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.397+0000 7f7a66806640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5f4c8c05-9231-4a4b-b76f-602253e33077", "id": 5}] 2024-03-25T17:29:27.398 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.397+0000 7f7a66806640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5f4c8c05-9231-4a4b-b76f-602253e33077", "id": 5}] 2024-03-25T17:29:27.398 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.397+0000 7f7a66806640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:27.398 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.397+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd new", "uuid": "5f4c8c05-9231-4a4b-b76f-602253e33077", "id": 5} v 0) v1 -- 0x7f7a600b5dc0 con 0x7f7a60142a20 2024-03-25T17:29:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.508+0000 7f7a5cff9640 1 -- 172.21.15.159:0/1729125524 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5f4c8c05-9231-4a4b-b76f-602253e33077", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2061194780 0 1988412176) 0x7f7a580047e0 con 0x7f7a60142a20 2024-03-25T17:29:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.508+0000 7f7a5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5f4c8c05-9231-4a4b-b76f-602253e33077", "id": 5}] 2024-03-25T17:29:27.509 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.508+0000 7f7a5cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:27.509 INFO:teuthology.orchestra.run.smithi159.stdout:5 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 10 librados: watch_flush enter 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 10 librados: watch_flush exit 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 20 client.34130.objecter shutdown clearing up homeless session... 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 10 client.34130.objecter successfully canceled tick 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 10 monclient: shutdown 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 >> v1:172.21.15.134:6790/0 conn(0x7f7a60142a20 legacy=0x7f7a601e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 shutdown_connections 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 >> 172.21.15.159:0/1729125524 conn(0x7f7a600668e0 msgr2=0x7f7a60066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 shutdown_connections 2024-03-25T17:29:27.511 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 1 -- 172.21.15.159:0/1729125524 wait complete. 2024-03-25T17:29:27.512 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.509+0000 7f7a66806640 1 librados: shutdown 2024-03-25T17:29:27.520 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 476003d3-e058-44fb-81a5-88ae20be2c3e 6 2024-03-25T17:29:27.563 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:27.562+0000 7f87c240f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:29:27.598 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:27.597+0000 7fed5a02b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T17:29:27.622 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.620+0000 7f721c939640 10 monclient: get_monmap_and_config 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.620+0000 7f721c939640 10 monclient: build_initial_monmap 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.620+0000 7f721c939640 10 monclient: monmap: 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:27.621306+0000 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:27.621306+0000 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:27.623 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.620+0000 7f721c939640 1 Processor -- start 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.620+0000 7f721c939640 1 -- start start 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.620+0000 7f721c939640 10 monclient: init 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 10 monclient(hunting): picked mon.noname-a con 0x7f72180b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 10 monclient(hunting): picked mon.noname-c con 0x7f7218142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 10 monclient(hunting): picked mon.noname-b con 0x7f72180a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72180a9d60 con 0x7f72180b4700 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721813ff10 con 0x7f7218142a20 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72180aa1f0 con 0x7f72180a96b0 2024-03-25T17:29:27.624 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:27.625 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.621+0000 7f721c939640 10 monclient(hunting): authenticate will time out at 5302.944288s 2024-03-25T17:29:27.625 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.622+0000 7f7215d9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7218142a20 0x7f721813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:44714/0 (socket says 172.21.15.159:44714) 2024-03-25T17:29:27.625 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.622+0000 7f7215d9b640 1 -- 172.21.15.159:0/1722266983 learned_addr learned my addr 172.21.15.159:0/1722266983 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:27.625 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.622+0000 7f721659c640 1 --1- 172.21.15.159:0/1722266983 >> v1:172.21.15.134:6789/0 conn(0x7f72180b4700 0x7f7218142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:45640/0 (socket says 172.21.15.159:45640) 2024-03-25T17:29:27.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.624+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843027149 0 0) 0x7f72180a9d60 con 0x7f72180b4700 2024-03-25T17:29:27.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.624+0000 7f721559a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.624+0000 7f721559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.624+0000 7f721559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.624+0000 7f721559a640 10 monclient(hunting): my global_id is 4152 2024-03-25T17:29:27.626 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.624+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7200003520 con 0x7f72180b4700 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.624+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006020573 0 0) 0x7f721813ff10 con 0x7f7218142a20 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): my global_id is 4144 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72180a9d60 con 0x7f7218142a20 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134891714 0 0) 0x7f72180aa1f0 con 0x7f72180a96b0 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 10 monclient(hunting): my global_id is 34133 2024-03-25T17:29:27.627 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.625+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f721813ff10 con 0x7f72180a96b0 2024-03-25T17:29:27.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3753976176 0 0) 0x7f7200003520 con 0x7f72180b4700 2024-03-25T17:29:27.628 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 >> v1:172.21.15.134:6790/0 conn(0x7f72180a96b0 legacy=0x7f721813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 >> v1:172.21.15.159:6789/0 conn(0x7f7218142a20 legacy=0x7f721813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 10 monclient: _finish_hunting 0 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 1 monclient: found mon.noname-a 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72180b4ae0 con 0x7f72180b4700 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 10 monclient: _finish_auth 0 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 10 monclient: _check_auth_tickets 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7208001690 con 0x7f72180b4700 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:27.629 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721559a640 10 monclient: dump: 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721c939640 5 monclient: authenticate success, global_id 4152 2024-03-25T17:29:27.630 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.627+0000 7f721c939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72080017f0 con 0x7f72180b4700 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721559a640 1 -- 172.21.15.159:0/1722266983 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7208001c00 con 0x7f72180b4700 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721c939640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721c939640 4 set_mon_vals no callback set 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.629+0000 7f721559a640 10 monclient: dump: 2024-03-25T17:29:27.634 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 10 monclient: shutdown 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 1 -- 172.21.15.159:0/1722266983 >> v1:172.21.15.134:6789/0 conn(0x7f72180b4700 legacy=0x7f7218142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 1 -- 172.21.15.159:0/1722266983 shutdown_connections 2024-03-25T17:29:27.635 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 1 -- 172.21.15.159:0/1722266983 >> 172.21.15.159:0/1722266983 conn(0x7f72180668e0 msgr2=0x7f7218066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 1 -- 172.21.15.159:0/1722266983 shutdown_connections 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 1 -- 172.21.15.159:0/1722266983 wait complete. 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 10 monclient: build_initial_monmap 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.630+0000 7f721c939640 10 monclient: monmap: 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:27.636 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 librados: starting msgr at 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 librados: starting objecter 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 Processor -- start 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 -- start start 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 librados: setting wanted keys 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 librados: calling monclient init 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient: init 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient(hunting): picked mon.noname-a con 0x7f72180a96b0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient(hunting): picked mon.noname-c con 0x7f72180b4700 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:27.637 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient(hunting): picked mon.noname-b con 0x7f7218142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721813ab40 con 0x7f72180a96b0 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72180ad9c0 con 0x7f7218142a20 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72181d7520 con 0x7f72180b4700 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.631+0000 7f721c939640 10 monclient(hunting): authenticate will time out at 5302.954560s 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f721659c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f72180a96b0 0x7f72180ab9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.159:45650/0 (socket says 172.21.15.159:45650) 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f721659c640 1 -- 172.21.15.159:0/609722732 learned_addr learned my addr 172.21.15.159:0/609722732 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f7206ffd640 10 client.?.objecter ms_handle_connect 0x7f72180a96b0 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f7206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f7206ffd640 10 client.?.objecter ms_handle_connect 0x7f7218142a20 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f7206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.638 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f7206ffd640 10 client.?.objecter ms_handle_connect 0x7f72180b4700 2024-03-25T17:29:27.639 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.632+0000 7f7206ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946459647 0 0) 0x7f721813ab40 con 0x7f72180a96b0 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71f8004eb0 con 0x7f72180a96b0 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300640850 0 0) 0x7f72180ad9c0 con 0x7f7218142a20 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.640 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.634+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f721813ab40 con 0x7f7218142a20 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.635+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647500932 0 0) 0x7f72181d7520 con 0x7f72180b4700 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.635+0000 7f7206ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.635+0000 7f7206ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.635+0000 7f7206ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.635+0000 7f7206ffd640 10 monclient(hunting): my global_id is 34136 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.635+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72180ad9c0 con 0x7f72180b4700 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.639+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1717981051 0 0) 0x7f721813ab40 con 0x7f7218142a20 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.639+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72181d7520 con 0x7f7218142a20 2024-03-25T17:29:27.641 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.639+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7210001bf0 con 0x7f7218142a20 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.639+0000 7f7206ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.639+0000 7f7206ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.639+0000 7f7206ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.642 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.643 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.639+0000 7f7206ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:27.643 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1087493272 0 0) 0x7f72180ad9c0 con 0x7f72180b4700 2024-03-25T17:29:27.643 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f721813ab40 con 0x7f72180b4700 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f720c0018d0 con 0x7f72180b4700 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.644 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2055666315 0 0) 0x7f71f8004eb0 con 0x7f72180a96b0 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72180ad9c0 con 0x7f72180a96b0 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7208001bc0 con 0x7f72180a96b0 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.640+0000 7f7206ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.645 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.646 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.646 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.646 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1612874706 0 0) 0x7f72181d7520 con 0x7f7218142a20 2024-03-25T17:29:27.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 >> v1:172.21.15.134:6790/0 conn(0x7f72180b4700 legacy=0x7f72181d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.647 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 >> v1:172.21.15.134:6789/0 conn(0x7f72180a96b0 legacy=0x7f72180ab9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 1 monclient: found mon.b 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72180adc30 con 0x7f7218142a20 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f7206ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 5 monclient: authenticate success, global_id 4147 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 10 monclient: _renew_subs 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72181d3fd0 con 0x7f7218142a20 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:27.648 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 10 monclient: _renew_subs 2024-03-25T17:29:27.649 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:27.649 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72181d82f0 con 0x7f7218142a20 2024-03-25T17:29:27.649 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.644+0000 7f721c939640 1 librados: init done 2024-03-25T17:29:27.649 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.645+0000 7f721c939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:27.649 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.645+0000 7f721c939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:27.649 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.645+0000 7f721c939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:27.649 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.645+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721804f730 con 0x7f7218142a20 2024-03-25T17:29:27.741 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.740+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7210002080 con 0x7f7218142a20 2024-03-25T17:29:27.741 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.740+0000 7f7206ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.740+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7210003b30 con 0x7f7218142a20 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.740+0000 7f7206ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 10 monclient: dump: 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.742 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7210004020 con 0x7f7218142a20 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.740+0000 7f7214d99640 4 set_mon_vals no callback set 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3689923322 0 0) 0x7f7210005830 con 0x7f7218142a20 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 10 client.4147.objecter ms_dispatch 0x7f7218000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7210017dc0 con 0x7f7218142a20 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:27.743 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.741+0000 7f7206ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:27.816 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.816+0000 7f721c939640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "476003d3-e058-44fb-81a5-88ae20be2c3e", "id": 6}] 2024-03-25T17:29:27.816 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.816+0000 7f721c939640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "476003d3-e058-44fb-81a5-88ae20be2c3e", "id": 6}] 2024-03-25T17:29:27.816 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.816+0000 7f721c939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:27.816 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.816+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd new", "uuid": "476003d3-e058-44fb-81a5-88ae20be2c3e", "id": 6} v 0) v1 -- 0x7f7218069f30 con 0x7f7218142a20 2024-03-25T17:29:27.851 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:27.850+0000 7f87c240f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:29:27.869 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.869+0000 7f7206ffd640 1 -- 172.21.15.159:0/609722732 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "476003d3-e058-44fb-81a5-88ae20be2c3e", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4219215752 0 1113811849) 0x7f7210005240 con 0x7f7218142a20 2024-03-25T17:29:27.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.869+0000 7f7206ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "476003d3-e058-44fb-81a5-88ae20be2c3e", "id": 6}] 2024-03-25T17:29:27.870 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.869+0000 7f7206ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:27.870 INFO:teuthology.orchestra.run.smithi159.stdout:6 2024-03-25T17:29:27.871 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.869+0000 7f721c939640 10 librados: watch_flush enter 2024-03-25T17:29:27.871 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.869+0000 7f721c939640 10 librados: watch_flush exit 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.869+0000 7f721c939640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.869+0000 7f721c939640 10 client.4147.objecter successfully canceled tick 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 10 monclient: shutdown 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 >> v1:172.21.15.159:6789/0 conn(0x7f7218142a20 legacy=0x7f72181d45a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 shutdown_connections 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 >> 172.21.15.159:0/609722732 conn(0x7f72180668e0 msgr2=0x7f72180b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 shutdown_connections 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 1 -- 172.21.15.159:0/609722732 wait complete. 2024-03-25T17:29:27.872 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.870+0000 7f721c939640 1 librados: shutdown 2024-03-25T17:29:27.881 DEBUG:teuthology.orchestra.run.smithi159:> sudo ceph --cluster ceph osd new 58eea41e-4a3f-4c11-86cb-a8d3d047599f 7 2024-03-25T17:29:27.888 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:27.887+0000 7fed5a02b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T17:29:27.905 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:27.904+0000 7f87c240f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:29:27.943 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:27.942+0000 7fed5a02b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T17:29:27.981 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.981+0000 7f3192b67640 10 monclient: get_monmap_and_config 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.981+0000 7f3192b67640 10 monclient: build_initial_monmap 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.981+0000 7f3192b67640 10 monclient: monmap: 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:29:27.982296+0000 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:29:27.982296+0000 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:27.982 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 1 Processor -- start 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 1 -- start start 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient: init 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient(hunting): picked mon.noname-a con 0x7f318c0b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient(hunting): picked mon.noname-c con 0x7f318c142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient(hunting): picked mon.noname-b con 0x7f318c0a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c0a9d60 con 0x7f318c0b4700 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c13ff10 con 0x7f318c142a20 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c0aa1f0 con 0x7f318c0a96b0 2024-03-25T17:29:27.984 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:27.985 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.982+0000 7f3192b67640 10 monclient(hunting): authenticate will time out at 5303.305261s 2024-03-25T17:29:27.985 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.983+0000 7f318bfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f318c142a20 0x7f318c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:44726/0 (socket says 172.21.15.159:44726) 2024-03-25T17:29:27.985 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.983+0000 7f318bfff640 1 -- 172.21.15.159:0/1286549379 learned_addr learned my addr 172.21.15.159:0/1286549379 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:27.986 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.985+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1209812736 0 0) 0x7f318c13ff10 con 0x7f318c142a20 2024-03-25T17:29:27.986 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.985+0000 7f318b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.986 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.985+0000 7f318b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.985+0000 7f318b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.985+0000 7f318b7fe640 10 monclient(hunting): my global_id is 4150 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.985+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f317c003520 con 0x7f318c142a20 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083109256 0 0) 0x7f318c0a9d60 con 0x7f318c0b4700 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): my global_id is 4158 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f318c13ff10 con 0x7f318c0b4700 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1973347814 0 0) 0x7f318c0aa1f0 con 0x7f318c0a96b0 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:27.987 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 10 monclient(hunting): my global_id is 34139 2024-03-25T17:29:27.988 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.986+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f318c0a9d60 con 0x7f318c0a96b0 2024-03-25T17:29:27.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729271407 0 0) 0x7f317c003520 con 0x7f318c142a20 2024-03-25T17:29:27.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 >> v1:172.21.15.134:6790/0 conn(0x7f318c0a96b0 legacy=0x7f318c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 >> v1:172.21.15.134:6789/0 conn(0x7f318c0b4700 legacy=0x7f318c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:27.989 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 1 monclient: found mon.noname-c 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c0b4ae0 con 0x7f318c142a20 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f3192b67640 5 monclient: authenticate success, global_id 4150 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3178001690 con 0x7f318c142a20 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f318b7fe640 10 monclient: dump: 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.990 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.991 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.988+0000 7f3192b67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31780017f0 con 0x7f318c142a20 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f318b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f318b7fe640 1 -- 172.21.15.159:0/1286549379 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3178001c00 con 0x7f318c142a20 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f3192b67640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f3192b67640 4 set_mon_vals no callback set 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f318b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f318b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f318b7fe640 10 monclient: dump: 2024-03-25T17:29:27.994 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f3192b67640 10 monclient: shutdown 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f3192b67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.990+0000 7f3192b67640 1 -- 172.21.15.159:0/1286549379 >> v1:172.21.15.159:6789/0 conn(0x7f318c142a20 legacy=0x7f318c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 1 -- 172.21.15.159:0/1286549379 shutdown_connections 2024-03-25T17:29:27.995 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 1 -- 172.21.15.159:0/1286549379 >> 172.21.15.159:0/1286549379 conn(0x7f318c0668e0 msgr2=0x7f318c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 1 -- 172.21.15.159:0/1286549379 shutdown_connections 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 1 -- 172.21.15.159:0/1286549379 wait complete. 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 10 monclient: build_initial_monmap 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 10 monclient: monmap: 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 0 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 0.000000 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:created 0.000000 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 1 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:27.996 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 1 librados: starting msgr at 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.991+0000 7f3192b67640 1 librados: starting objecter 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 1 Processor -- start 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 1 -- start start 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 1 librados: setting wanted keys 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 1 librados: calling monclient init 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient: init 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient(hunting): picked mon.noname-c con 0x7f318c0a96b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient(hunting): picked mon.noname-a con 0x7f318c0b4700 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient(hunting): picked mon.noname-b con 0x7f318c142a20 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c1d7500 con 0x7f318c0b4700 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c13cce0 con 0x7f318c142a20 2024-03-25T17:29:27.997 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f318c1d8130 con 0x7f318c0a96b0 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.992+0000 7f3192b67640 10 monclient(hunting): authenticate will time out at 5303.315451s 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f3191104640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f318c142a20 0x7f318c1e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.159:44738/0 (socket says 172.21.15.159:44738) 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f3191104640 1 -- 172.21.15.159:0/3136701387 learned_addr learned my addr 172.21.15.159:0/3136701387 (peer_addr_for_me v1:172.21.15.159:0/0) 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f31897fa640 10 client.?.objecter ms_handle_connect 0x7f318c142a20 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f31897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f31897fa640 10 client.?.objecter ms_handle_connect 0x7f318c0b4700 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f31897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f31897fa640 10 client.?.objecter ms_handle_connect 0x7f318c0a96b0 2024-03-25T17:29:27.998 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.993+0000 7f31897fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:27.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.995+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783413869 0 0) 0x7f318c13cce0 con 0x7f318c142a20 2024-03-25T17:29:27.999 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.995+0000 7f31897fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.995+0000 7f31897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.995+0000 7f31897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.995+0000 7f31897fa640 10 monclient(hunting): my global_id is 4153 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.995+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3170004eb0 con 0x7f318c142a20 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206904401 0 0) 0x7f318c1d7500 con 0x7f318c0b4700 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): my global_id is 4161 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f318c13cce0 con 0x7f318c0b4700 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402704260 0 0) 0x7f318c1d8130 con 0x7f318c0a96b0 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:28.000 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:28.001 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:28.001 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 10 monclient(hunting): my global_id is 34142 2024-03-25T17:29:28.001 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:27.996+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f318c1d7500 con 0x7f318c0a96b0 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4235131971 0 0) 0x7f3170004eb0 con 0x7f318c142a20 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f318c1d8130 con 0x7f318c142a20 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3184001d10 con 0x7f318c142a20 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 10 monclient(hunting): dump: 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.003 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2677278027 0 0) 0x7f318c1d7500 con 0x7f318c0a96b0 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3170004eb0 con 0x7f318c0a96b0 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3180001930 con 0x7f318c0a96b0 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.004 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.001+0000 7f31897fa640 10 monclient(hunting): dump: 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:28.005 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.002+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1130113542 0 0) 0x7f318c13cce0 con 0x7f318c0b4700 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.002+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f318c1d7500 con 0x7f318c0b4700 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.002+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3178001b00 con 0x7f318c0b4700 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.002+0000 7f31897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.002+0000 7f31897fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.002+0000 7f31897fa640 10 monclient(hunting): dump: 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.006 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.007 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.007 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:28.010 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1737670482 0 0) 0x7f318c1d8130 con 0x7f318c142a20 2024-03-25T17:29:28.010 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 >> v1:172.21.15.134:6790/0 conn(0x7f318c0a96b0 legacy=0x7f318c1d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:28.010 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 >> v1:172.21.15.134:6789/0 conn(0x7f318c0b4700 legacy=0x7f318c1d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:28.010 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 10 monclient: _finish_hunting 0 2024-03-25T17:29:28.010 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 1 monclient: found mon.b 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.004+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318c1d82c0 con 0x7f318c142a20 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f31897fa640 10 monclient: _finish_auth 0 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f31897fa640 10 monclient: _check_auth_tickets 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f31897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 5 monclient: authenticate success, global_id 4153 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 10 monclient: _renew_subs 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318c1d3fd0 con 0x7f318c142a20 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 10 monclient: _renew_subs 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f318c1d4f40 con 0x7f318c142a20 2024-03-25T17:29:28.011 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.005+0000 7f3192b67640 1 librados: init done 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.006+0000 7f3192b67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.006+0000 7f3192b67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.006+0000 7f3192b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.006+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318c1d7740 con 0x7f318c142a20 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3184003990 con 0x7f318c142a20 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3184003bc0 con 0x7f318c142a20 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 10 monclient: dump: 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:epoch 1 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.012 INFO:teuthology.orchestra.run.smithi159.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:election_strategy: 3 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3184004170 con 0x7f318c142a20 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3398249441 0 0) 0x7f3184006a70 con 0x7f318c142a20 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 10 client.4153.objecter ms_dispatch 0x7f318c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f31897fa640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.008+0000 7f318affd640 4 set_mon_vals no callback set 2024-03-25T17:29:28.013 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.009+0000 7f31897fa640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T17:29:28.026 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:28.025+0000 7f87c240f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:29:28.064 INFO:tasks.ceph.mgr.x.smithi159.stderr:2024-03-25T17:29:28.064+0000 7fed5a02b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T17:29:28.102 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.101+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3184006520 con 0x7f318c142a20 2024-03-25T17:29:28.153 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.101+0000 7f31897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:28.153 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.101+0000 7f31897fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:28.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.173+0000 7f3192b67640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "58eea41e-4a3f-4c11-86cb-a8d3d047599f", "id": 7}] 2024-03-25T17:29:28.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.173+0000 7f3192b67640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "58eea41e-4a3f-4c11-86cb-a8d3d047599f", "id": 7}] 2024-03-25T17:29:28.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.173+0000 7f3192b67640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.174 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.173+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd new", "uuid": "58eea41e-4a3f-4c11-86cb-a8d3d047599f", "id": 7} v 0) v1 -- 0x7f318c04f730 con 0x7f318c142a20 2024-03-25T17:29:28.267 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.266+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1651191815 0 0) 0x7f3184006760 con 0x7f318c142a20 2024-03-25T17:29:28.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.491+0000 7f31897fa640 1 -- 172.21.15.159:0/3136701387 <== mon.1 v1:172.21.15.159:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "58eea41e-4a3f-4c11-86cb-a8d3d047599f", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 530625312 0 1371665406) 0x7f3184016080 con 0x7f318c142a20 2024-03-25T17:29:28.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.491+0000 7f31897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "58eea41e-4a3f-4c11-86cb-a8d3d047599f", "id": 7}] 2024-03-25T17:29:28.492 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.491+0000 7f31897fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:28.492 INFO:teuthology.orchestra.run.smithi159.stdout:7 2024-03-25T17:29:28.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 10 librados: watch_flush enter 2024-03-25T17:29:28.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 10 librados: watch_flush exit 2024-03-25T17:29:28.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T17:29:28.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 10 client.4153.objecter successfully canceled tick 2024-03-25T17:29:28.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 10 monclient: shutdown 2024-03-25T17:29:28.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:28.493 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 >> v1:172.21.15.159:6789/0 conn(0x7f318c142a20 legacy=0x7f318c1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:28.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 shutdown_connections 2024-03-25T17:29:28.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 >> 172.21.15.159:0/3136701387 conn(0x7f318c0668e0 msgr2=0x7f318c13f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:28.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 shutdown_connections 2024-03-25T17:29:28.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 1 -- 172.21.15.159:0/3136701387 wait complete. 2024-03-25T17:29:28.494 INFO:teuthology.orchestra.run.smithi159.stderr:2024-03-25T17:29:28.492+0000 7f3192b67640 1 librados: shutdown 2024-03-25T17:29:28.503 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.503 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T17:29:28.503 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T17:29:28.505 INFO:tasks.ceph.osd.0:Started 2024-03-25T17:29:28.505 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.506 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T17:29:28.506 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T17:29:28.508 INFO:tasks.ceph.osd.1:Started 2024-03-25T17:29:28.508 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.508 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T17:29:28.508 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T17:29:28.510 INFO:tasks.ceph.osd.2:Started 2024-03-25T17:29:28.510 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.510 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T17:29:28.511 DEBUG:teuthology.orchestra.run.smithi134:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T17:29:28.513 INFO:tasks.ceph.osd.3:Started 2024-03-25T17:29:28.513 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.513 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T17:29:28.513 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T17:29:28.545 INFO:tasks.ceph.osd.4:Started 2024-03-25T17:29:28.546 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.546 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T17:29:28.546 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T17:29:28.548 INFO:tasks.ceph.osd.5:Started 2024-03-25T17:29:28.548 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.548 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T17:29:28.548 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T17:29:28.550 INFO:tasks.ceph.osd.6:Started 2024-03-25T17:29:28.551 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T17:29:28.551 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T17:29:28.551 DEBUG:teuthology.orchestra.run.smithi159:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T17:29:28.553 INFO:tasks.ceph.osd.7:Started 2024-03-25T17:29:28.553 DEBUG:teuthology.orchestra.run.smithi134:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:28.716 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.716+0000 7f5fa3132640 10 monclient: get_monmap_and_config 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.716+0000 7f5fa3132640 10 monclient: build_initial_monmap 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.716+0000 7f5fa3132640 10 monclient: monmap: 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:29:28.717444+0000 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:29:28.717444+0000 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:28.717 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.717+0000 7f5fa3132640 1 Processor -- start 2024-03-25T17:29:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.717+0000 7f5fa3132640 1 -- start start 2024-03-25T17:29:28.718 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.717+0000 7f5fa3132640 10 monclient: init 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.718+0000 7f5fa3132640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.718+0000 7f5fa3132640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.718+0000 7f5fa3132640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9c0a9920 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.718+0000 7f5fa3132640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9c0b45a0 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa3132640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9c146450 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa3132640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c06b100 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa3132640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0b2940 con 0x7f5f9c146450 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa3132640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c13fec0 con 0x7f5f9c0a9920 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa3132640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa3132640 10 monclient(hunting): authenticate will time out at 5321.982213s 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa0ece640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f5f9c0a9920 0x7f5f9c13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:52794/0 (socket says 172.21.15.134:52794) 2024-03-25T17:29:28.720 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.719+0000 7f5fa0ece640 1 -- 172.21.15.134:0/299257051 learned_addr learned my addr 172.21.15.134:0/299257051 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164818161 0 0) 0x7f5f9c0b2940 con 0x7f5f9c146450 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): my global_id is 4174 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f8c003520 con 0x7f5f9c146450 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047443241 0 0) 0x7f5f9c13fec0 con 0x7f5f9c0a9920 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 10 monclient(hunting): my global_id is 34154 2024-03-25T17:29:28.725 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.723+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9c0b2940 con 0x7f5f9c0a9920 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 76276433 0 0) 0x7f5f8c003520 con 0x7f5f9c146450 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 >> v1:172.21.15.134:6790/0 conn(0x7f5f9c0a9920 legacy=0x7f5f9c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 >> v1:172.21.15.134:6789/0 conn(0x7f5f9c0b45a0 legacy=0x7f5f9c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 1 monclient: found mon.noname-c 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c140050 con 0x7f5f9c146450 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:28.729 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5fa3132640 5 monclient: authenticate success, global_id 4174 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5fa3132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5f94001a10 con 0x7f5f9c146450 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5f9b7fe640 10 monclient: dump: 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.730 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.731 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:28.731 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.728+0000 7f5fa3132640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f94001c60 con 0x7f5f9c146450 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5f9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5f9b7fe640 1 -- 172.21.15.134:0/299257051 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5f94001e50 con 0x7f5f9c146450 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5f9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5f9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5f9b7fe640 10 monclient: dump: 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.732 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5fa3132640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:28.733 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.731+0000 7f5fa3132640 4 set_mon_vals no callback set 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.732+0000 7f5fa3132640 10 monclient: shutdown 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.732+0000 7f5fa3132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.732+0000 7f5fa3132640 1 -- 172.21.15.134:0/299257051 >> v1:172.21.15.159:6789/0 conn(0x7f5f9c146450 legacy=0x7f5f9c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.732+0000 7f5fa3132640 1 -- 172.21.15.134:0/299257051 shutdown_connections 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.732+0000 7f5fa3132640 1 -- 172.21.15.134:0/299257051 >> 172.21.15.134:0/299257051 conn(0x7f5f9c0b7f40 msgr2=0x7f5f9c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.732+0000 7f5fa3132640 1 -- 172.21.15.134:0/299257051 shutdown_connections 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.732+0000 7f5fa3132640 1 -- 172.21.15.134:0/299257051 wait complete. 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.733+0000 7f5fa3132640 10 monclient: build_initial_monmap 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.733+0000 7f5fa3132640 10 monclient: monmap: 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 0 2024-03-25T17:29:28.735 INFO:teuthology.orchestra.run.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 0.000000 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:created 0.000000 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 1 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.734+0000 7f5fa3132640 1 librados: starting msgr at 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.734+0000 7f5fa3132640 1 librados: starting objecter 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.734+0000 7f5fa3132640 1 Processor -- start 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.734+0000 7f5fa3132640 1 -- start start 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.734+0000 7f5fa3132640 1 librados: setting wanted keys 2024-03-25T17:29:28.736 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.734+0000 7f5fa3132640 1 librados: calling monclient init 2024-03-25T17:29:28.737 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.734+0000 7f5fa3132640 10 monclient: init 2024-03-25T17:29:28.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.735+0000 7f5fa3132640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:28.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.735+0000 7f5fa3132640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:28.738 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.735+0000 7f5fa3132640 10 monclient(hunting): picked mon.noname-c con 0x7f5f9c0a9920 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.735+0000 7f5fa3132640 10 monclient(hunting): picked mon.noname-b con 0x7f5f9c0b45a0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.735+0000 7f5fa3132640 10 monclient(hunting): picked mon.noname-a con 0x7f5f9c146450 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.735+0000 7f5fa3132640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c144020 con 0x7f5f9c146450 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.735+0000 7f5fa3132640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c140290 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5fa3132640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c1357d0 con 0x7f5f9c0a9920 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5fa3132640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5fa3132640 10 monclient(hunting): authenticate will time out at 5321.998915s 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5fa0ece640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f5f9c0a9920 0x7f5f9c1350c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:52806/0 (socket says 172.21.15.134:52806) 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5fa0ece640 1 -- 172.21.15.134:0/1421556608 learned_addr learned my addr 172.21.15.134:0/1421556608 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5f997fa640 10 client.?.objecter ms_handle_connect 0x7f5f9c0b45a0 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5f997fa640 10 client.?.objecter ms_handle_connect 0x7f5f9c0a9920 2024-03-25T17:29:28.739 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.736+0000 7f5f997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067855982 0 0) 0x7f5f9c140290 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): my global_id is 4177 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f80004eb0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054662680 0 0) 0x7f5f9c1357d0 con 0x7f5f9c0a9920 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): _negotiate 2024-03-25T17:29:28.741 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 10 monclient(hunting): my global_id is 34157 2024-03-25T17:29:28.742 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.740+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9c140290 con 0x7f5f9c0a9920 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.747+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919472698 0 0) 0x7f5f80004eb0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.747+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f9c1357d0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.747+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5f88001930 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.747+0000 7f5f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.747+0000 7f5f997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.747+0000 7f5f997fa640 10 monclient(hunting): dump: 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.748 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.749 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:28.749 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.747+0000 7f5f997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.748+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1776636310 0 0) 0x7f5f9c140290 con 0x7f5f9c0a9920 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.748+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f80004eb0 con 0x7f5f9c0a9920 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.748+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5f900019f0 con 0x7f5f9c0a9920 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.748+0000 7f5f997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.748+0000 7f5f997fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.748+0000 7f5f997fa640 10 monclient(hunting): dump: 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.750 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.751 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.751 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:28.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3202531552 0 0) 0x7f5f9c1357d0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 >> v1:172.21.15.134:6790/0 conn(0x7f5f9c0a9920 legacy=0x7f5f9c1350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:28.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 >> v1:172.21.15.134:6789/0 conn(0x7f5f9c146450 legacy=0x7f5f9c1407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:28.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 10 monclient: _finish_hunting 0 2024-03-25T17:29:28.753 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 1 monclient: found mon.b 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c135960 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 10 monclient: _finish_auth 0 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 10 monclient: _check_auth_tickets 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5f997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5fa3132640 5 monclient: authenticate success, global_id 4177 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5fa3132640 10 monclient: _renew_subs 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5fa3132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.752+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c1db0b0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.753+0000 7f5fa3132640 10 client.4177.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.753+0000 7f5fa3132640 10 monclient: _renew_subs 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.753+0000 7f5fa3132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.754 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.753+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c13bb70 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.753+0000 7f5fa3132640 1 librados: init done 2024-03-25T17:29:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.754+0000 7f5fa3132640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.754+0000 7f5fa3132640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.754+0000 7f5fa3132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:28.755 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.754+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c135af0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f880035f0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5f88003900 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f997fa640 10 monclient: dump: 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:epoch 1 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.935 INFO:teuthology.orchestra.run.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f9affd640 4 set_mon_vals no callback set 2024-03-25T17:29:28.936 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.934+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1651191815 0 0) 0x7f5f8800e9e0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f5f880144c0 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 10 client.4177.objecter ms_dispatch 0x7f5f9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 3 client.4177.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 3 client.4177.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 20 client.4177.objecter dump_active .. 0 homeless 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f5f88013780 con 0x7f5f9c0b45a0 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:28.940 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:28.939+0000 7f5f997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.062+0000 7f5fa3132640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.062+0000 7f5fa3132640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.062+0000 7f5fa3132640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:29.063 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.062+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f9c04f730 con 0x7f5f9c0b45a0 2024-03-25T17:29:29.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.273+0000 7f5f997fa640 1 -- 172.21.15.134:0/1421556608 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f5f88011f60 con 0x7f5f9c0b45a0 2024-03-25T17:29:29.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.273+0000 7f5f997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:29.275 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.273+0000 7f5f997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:29.275 INFO:teuthology.orchestra.run.smithi134.stdout: 2024-03-25T17:29:29.275 INFO:teuthology.orchestra.run.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:29.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.274+0000 7f5fa3132640 10 librados: watch_flush enter 2024-03-25T17:29:29.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.274+0000 7f5fa3132640 10 librados: watch_flush exit 2024-03-25T17:29:29.276 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.274+0000 7f5fa3132640 20 client.4177.objecter shutdown clearing up homeless session... 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.274+0000 7f5fa3132640 10 client.4177.objecter successfully canceled tick 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.274+0000 7f5fa3132640 10 monclient: shutdown 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.274+0000 7f5fa3132640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.274+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 >> v1:172.21.15.159:6789/0 conn(0x7f5f9c0b45a0 legacy=0x7f5f9c1da9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.275+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 shutdown_connections 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.275+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 >> 172.21.15.134:0/1421556608 conn(0x7f5f9c0b7f40 msgr2=0x7f5f9c0b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.275+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 shutdown_connections 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.275+0000 7f5fa3132640 1 -- 172.21.15.134:0/1421556608 wait complete. 2024-03-25T17:29:29.277 INFO:teuthology.orchestra.run.smithi134.stderr:2024-03-25T17:29:29.275+0000 7f5fa3132640 1 librados: shutdown 2024-03-25T17:29:29.291 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T17:29:29.291 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T17:29:29.847 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:29.845+0000 7f8762c26640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T17:29:29.854 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:29:29.852+0000 7f8762c26640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T17:29:30.096 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:30.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.240+0000 7fb74c80f640 10 monclient: get_monmap_and_config 2024-03-25T17:29:30.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.240+0000 7fb74c80f640 10 monclient: build_initial_monmap 2024-03-25T17:29:30.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.240+0000 7fb74c80f640 10 monclient: monmap: 2024-03-25T17:29:30.242 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:30.242 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:29:30.242878+0000 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:29:30.242878+0000 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:30.243 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.241+0000 7fb74c80f640 1 Processor -- start 2024-03-25T17:29:30.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.241+0000 7fb74c80f640 1 -- start start 2024-03-25T17:29:30.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.241+0000 7fb74c80f640 10 monclient: init 2024-03-25T17:29:30.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.242+0000 7fb74c80f640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:30.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.242+0000 7fb74c80f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:30.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.242+0000 7fb74c80f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7480a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:30.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.242+0000 7fb74c80f640 10 monclient(hunting): picked mon.noname-a con 0x7fb74813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.242+0000 7fb74c80f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7481430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.242+0000 7fb74c80f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb748142520 con 0x7fb74813f710 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.243+0000 7fb74c80f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7480b28a0 con 0x7fb7480a98f0 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.243+0000 7fb74c80f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7480b2a30 con 0x7fb7481430c0 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.243+0000 7fb74c80f640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.243+0000 7fb74c80f640 10 monclient(hunting): authenticate will time out at 5323.506793s 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.244+0000 7fb746d9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fb7481430c0 0x7fb74814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:55374/0 (socket says 172.21.15.134:55374) 2024-03-25T17:29:30.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.244+0000 7fb746d9d640 1 -- 172.21.15.134:0/3874907951 learned_addr learned my addr 172.21.15.134:0/3874907951 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:30.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.248+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674530566 0 0) 0x7fb7480b2a30 con 0x7fb7481430c0 2024-03-25T17:29:30.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.248+0000 7fb74559a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:30.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.248+0000 7fb74559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:30.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.248+0000 7fb74559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:30.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.248+0000 7fb74559a640 10 monclient(hunting): my global_id is 34160 2024-03-25T17:29:30.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.248+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb730003520 con 0x7fb7481430c0 2024-03-25T17:29:30.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342719501 0 0) 0x7fb748142520 con 0x7fb74813f710 2024-03-25T17:29:30.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:30.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:30.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:30.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): my global_id is 4167 2024-03-25T17:29:30.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7480b2a30 con 0x7fb74813f710 2024-03-25T17:29:30.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138110481 0 0) 0x7fb7480b28a0 con 0x7fb7480a98f0 2024-03-25T17:29:30.254 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): _negotiate 2024-03-25T17:29:30.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:30.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:30.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 10 monclient(hunting): my global_id is 4180 2024-03-25T17:29:30.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.249+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb748142520 con 0x7fb7480a98f0 2024-03-25T17:29:30.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3824081302 0 0) 0x7fb730003520 con 0x7fb7481430c0 2024-03-25T17:29:30.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 >> v1:172.21.15.159:6789/0 conn(0x7fb7480a98f0 legacy=0x7fb7480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:30.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 >> v1:172.21.15.134:6789/0 conn(0x7fb74813f710 legacy=0x7fb7481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 10 monclient: _finish_hunting 0 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 1 monclient: found mon.noname-b 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7480b44c0 con 0x7fb7481430c0 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 10 monclient: _finish_auth 0 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 10 monclient: _check_auth_tickets 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb740001a10 con 0x7fb7481430c0 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74559a640 10 monclient: dump: 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:30.257 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74c80f640 5 monclient: authenticate success, global_id 34160 2024-03-25T17:29:30.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.253+0000 7fb74c80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:30.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb740001c30 con 0x7fb7481430c0 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74c80f640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74c80f640 4 set_mon_vals no callback set 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74559a640 1 -- 172.21.15.134:0/3874907951 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb740001fc0 con 0x7fb7481430c0 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.256+0000 7fb74559a640 10 monclient: dump: 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:30.261 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.257+0000 7fb74c80f640 10 monclient: shutdown 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.257+0000 7fb74c80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.257+0000 7fb74c80f640 1 -- 172.21.15.134:0/3874907951 >> v1:172.21.15.134:6790/0 conn(0x7fb7481430c0 legacy=0x7fb74814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.257+0000 7fb74c80f640 1 -- 172.21.15.134:0/3874907951 shutdown_connections 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.257+0000 7fb74c80f640 1 -- 172.21.15.134:0/3874907951 >> 172.21.15.134:0/3874907951 conn(0x7fb7480b7ea0 msgr2=0x7fb7480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.257+0000 7fb74c80f640 1 -- 172.21.15.134:0/3874907951 shutdown_connections 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.257+0000 7fb74c80f640 1 -- 172.21.15.134:0/3874907951 wait complete. 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.258+0000 7fb74c80f640 10 monclient: build_initial_monmap 2024-03-25T17:29:30.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.258+0000 7fb74c80f640 10 monclient: monmap: 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.258+0000 7fb74c80f640 1 librados: starting msgr at 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.258+0000 7fb74c80f640 1 librados: starting objecter 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.259+0000 7fb74c80f640 1 Processor -- start 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.259+0000 7fb74c80f640 1 -- start start 2024-03-25T17:29:30.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.259+0000 7fb74c80f640 1 librados: setting wanted keys 2024-03-25T17:29:30.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.259+0000 7fb74c80f640 1 librados: calling monclient init 2024-03-25T17:29:30.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.259+0000 7fb74c80f640 10 monclient: init 2024-03-25T17:29:30.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:30.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:30.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 10 monclient(hunting): picked mon.noname-c con 0x7fb7480a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 10 monclient(hunting): picked mon.noname-a con 0x7fb74813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 10 monclient(hunting): picked mon.noname-b con 0x7fb7481430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb748138d70 con 0x7fb74813f710 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74659c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fb7480a98f0 0x7fb748137da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:55384/0 (socket says 172.21.15.134:55384) 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74659c640 1 -- 172.21.15.134:0/1126645209 learned_addr learned my addr 172.21.15.134:0/1126645209 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7481dc070 con 0x7fb7481430c0 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb748138fe0 con 0x7fb7480a98f0 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.260+0000 7fb74c80f640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.261+0000 7fb74c80f640 10 monclient(hunting): authenticate will time out at 5323.524812s 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.261+0000 7fb736ffd640 10 client.?.objecter ms_handle_connect 0x7fb7480a98f0 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.261+0000 7fb736ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.261+0000 7fb736ffd640 10 client.?.objecter ms_handle_connect 0x7fb74813f710 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.261+0000 7fb736ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:30.268 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.261+0000 7fb736ffd640 10 client.?.objecter ms_handle_connect 0x7fb7481430c0 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.261+0000 7fb736ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.263+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519247386 0 0) 0x7fb748138d70 con 0x7fb74813f710 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.263+0000 7fb736ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.263+0000 7fb736ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.263+0000 7fb736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.263+0000 7fb736ffd640 10 monclient(hunting): my global_id is 4170 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.263+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb728004eb0 con 0x7fb74813f710 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499721831 0 0) 0x7fb748138fe0 con 0x7fb7480a98f0 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 10 monclient(hunting): my global_id is 34163 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb748138d70 con 0x7fb7480a98f0 2024-03-25T17:29:30.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545836040 0 0) 0x7fb7481dc070 con 0x7fb7481430c0 2024-03-25T17:29:30.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:30.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:30.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.264+0000 7fb736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:30.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.265+0000 7fb736ffd640 10 monclient(hunting): my global_id is 4183 2024-03-25T17:29:30.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.265+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb748138fe0 con 0x7fb7481430c0 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.270+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121968076 0 0) 0x7fb728004eb0 con 0x7fb74813f710 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.270+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7481dc070 con 0x7fb74813f710 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.270+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb73c001930 con 0x7fb74813f710 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.270+0000 7fb736ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.270+0000 7fb736ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.270+0000 7fb736ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:30.272 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.270+0000 7fb736ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:30.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.271+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 780371173 0 0) 0x7fb748138d70 con 0x7fb7480a98f0 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.271+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb728004eb0 con 0x7fb7480a98f0 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb738001a50 con 0x7fb7480a98f0 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:30.275 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1413499822 0 0) 0x7fb748138fe0 con 0x7fb7481430c0 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb748138d70 con 0x7fb7481430c0 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb740001da0 con 0x7fb7481430c0 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.272+0000 7fb736ffd640 10 monclient(hunting): dump: 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:30.276 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:30.277 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:30.277 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:30.277 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2923988424 0 0) 0x7fb728004eb0 con 0x7fb7480a98f0 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 >> v1:172.21.15.159:6789/0 conn(0x7fb7481430c0 legacy=0x7fb7481db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 >> v1:172.21.15.134:6789/0 conn(0x7fb74813f710 legacy=0x7fb7481db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 1 monclient: found mon.c 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7481392c0 con 0x7fb7480a98f0 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:30.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb74c80f640 5 monclient: authenticate success, global_id 34163 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb74c80f640 10 monclient: _renew_subs 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb74c80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.276+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7481395a0 con 0x7fb7480a98f0 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.277+0000 7fb74c80f640 10 client.34163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.277+0000 7fb74c80f640 10 monclient: _renew_subs 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.277+0000 7fb74c80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.277+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7481384b0 con 0x7fb7480a98f0 2024-03-25T17:29:30.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.277+0000 7fb74c80f640 1 librados: init done 2024-03-25T17:29:30.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.278+0000 7fb74c80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:30.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.278+0000 7fb74c80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:30.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.278+0000 7fb74c80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:30.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.278+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb74814f320 con 0x7fb7480a98f0 2024-03-25T17:29:30.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.718+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb738001bb0 con 0x7fb7480a98f0 2024-03-25T17:29:30.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.718+0000 7fb736ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:30.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.718+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb738003ae0 con 0x7fb7480a98f0 2024-03-25T17:29:30.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.718+0000 7fb736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:30.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.718+0000 7fb736ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:30.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.718+0000 7fb736ffd640 10 monclient: dump: 2024-03-25T17:29:30.719 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:30.720 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:30.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.718+0000 7fb744d99640 4 set_mon_vals no callback set 2024-03-25T17:29:30.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.719+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 650736104 0 0) 0x7fb738003d00 con 0x7fb7480a98f0 2024-03-25T17:29:30.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.720+0000 7fb745d9b640 10 monclient: get_auth_request con 0x7fb72806e090 auth_method 0 2024-03-25T17:29:30.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.720+0000 7fb736ffd640 10 client.34163.objecter ms_handle_connect 0x7fb72806e090 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fb7380547a0 con 0x7fb7480a98f0 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 10 client.34163.objecter ms_dispatch 0x7fb748000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 3 client.34163.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 3 client.34163.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 20 client.34163.objecter dump_active .. 0 homeless 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7380502b0 con 0x7fb7480a98f0 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:30.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:30.725+0000 7fb736ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:31.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.061+0000 7fb74c80f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:31.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.061+0000 7fb74c80f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:31.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.061+0000 7fb74c80f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:31.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.061+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7480627b0 con 0x7fb7480a98f0 2024-03-25T17:29:31.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.066+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fb73804fd60 con 0x7fb7480a98f0 2024-03-25T17:29:31.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.194+0000 7fb736ffd640 1 -- 172.21.15.134:0/1126645209 <== mon.2 v1:172.21.15.134:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fb738003940 con 0x7fb7480a98f0 2024-03-25T17:29:31.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.194+0000 7fb736ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:31.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.194+0000 7fb736ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:31.195 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:29:31.195 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 10 librados: watch_flush enter 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 10 librados: watch_flush exit 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 20 client.34163.objecter shutdown clearing up homeless session... 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 10 client.34163.objecter successfully canceled tick 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 >> v1:172.21.15.134:6800/610837766 conn(0x7fb72806e090 legacy=0x7fb728070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 10 monclient: shutdown 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 >> v1:172.21.15.134:6790/0 conn(0x7fb7480a98f0 legacy=0x7fb748137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:31.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 shutdown_connections 2024-03-25T17:29:31.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 >> 172.21.15.134:0/1126645209 conn(0x7fb7480b7ea0 msgr2=0x7fb74813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:31.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 shutdown_connections 2024-03-25T17:29:31.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 1 -- 172.21.15.134:0/1126645209 wait complete. 2024-03-25T17:29:31.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:31.197+0000 7fb74c80f640 1 librados: shutdown 2024-03-25T17:29:31.214 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:35.382 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:06.769 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.444 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:06.831 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.453 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:06.836 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.465 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:06.841 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.513 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:06.896 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.529 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:06.904 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.630 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:07.012 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.687 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:07.070 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.714 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:07.119 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.755 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:07.159 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.776 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:07.181 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.816 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:07.221 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.819 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:07.237 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.898 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:07.315 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:35.987 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-25T17:29:35.981+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.007 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:07.409 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.066 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:07.469 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.079 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-25T17:29:36.070+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.085 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-25T17:29:36.079+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.242 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-25T17:29:36.236+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.317 INFO:tasks.ceph.osd.0.smithi134.stderr:2024-03-25T17:29:36.311+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.408 INFO:tasks.ceph.osd.3.smithi134.stderr:2024-03-25T17:29:36.402+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.463 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:07.850 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.496 INFO:tasks.ceph.osd.1.smithi134.stderr:2024-03-25T17:29:36.488+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.561 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:07.944 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.577 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:07.952 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.655 INFO:tasks.ceph.osd.2.smithi134.stderr:2024-03-25T17:29:36.646+0000 5dc3480 -1 Falling back to public interface 2024-03-25T17:29:36.709 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:08.092 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.809 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:08.214 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.828 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:08.216 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.902 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:08.306 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.925 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:08.308 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.955 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:08.331 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:36.957 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:08.375 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.074 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:08.457 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.106 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:08.509 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.140 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:08.527 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.205 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:08.610 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.274 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:08.678 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.274 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:08.649 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.284 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:08.667 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.327 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:08.744 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.410 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:08.797 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.429 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:08.812 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.469 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:08.871 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.521 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:08.926 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.539 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:08.914 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.576 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:08.958 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.627 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.032 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.674 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.061 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.684 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:09.101 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.717 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.099 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.780 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:09.182 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.802 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:09.206 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.804 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.179 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.873 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:09.255 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.916 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.320 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.938 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.325 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:37.971 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:09.388 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.008 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.391 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.019 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:38.067 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:09.469 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.068 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.443 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.079 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:09.484 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.109+0000 7f32da7f4640 10 monclient: get_monmap_and_config 2024-03-25T17:29:38.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.109+0000 7f32da7f4640 10 monclient: build_initial_monmap 2024-03-25T17:29:38.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.110+0000 7f32da7f4640 10 monclient: monmap: 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:29:38.110979+0000 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:29:38.110979+0000 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:38.111 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.112 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.110+0000 7f32da7f4640 1 Processor -- start 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.110+0000 7f32da7f4640 1 -- start start 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.110+0000 7f32da7f4640 10 monclient: init 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f32d40b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f32d40b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f32d414be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40af940 con 0x7f32d414be90 2024-03-25T17:29:38.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40b0a70 con 0x7f32d40b4d60 2024-03-25T17:29:38.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d40b0d50 con 0x7f32d40b1730 2024-03-25T17:29:38.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:38.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32da7f4640 10 monclient(hunting): authenticate will time out at 5331.374159s 2024-03-25T17:29:38.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32d37fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f32d40b1730 0x7f32d40b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:55416/0 (socket says 172.21.15.134:55416) 2024-03-25T17:29:38.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.111+0000 7f32d37fe640 1 -- 172.21.15.134:0/3843431007 learned_addr learned my addr 172.21.15.134:0/3843431007 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:38.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782390197 0 0) 0x7f32d40b0d50 con 0x7f32d40b1730 2024-03-25T17:29:38.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:38.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): my global_id is 34190 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32bc003520 con 0x7f32d40b1730 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918565533 0 0) 0x7f32d40b0a70 con 0x7f32d40b4d60 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): my global_id is 4210 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d40b0d50 con 0x7f32d40b4d60 2024-03-25T17:29:38.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121328948 0 0) 0x7f32d40af940 con 0x7f32d414be90 2024-03-25T17:29:38.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:29:38.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:38.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:38.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 10 monclient(hunting): my global_id is 4197 2024-03-25T17:29:38.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.114+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d40b0a70 con 0x7f32d414be90 2024-03-25T17:29:38.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1840161515 0 0) 0x7f32bc003520 con 0x7f32d40b1730 2024-03-25T17:29:38.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 >> v1:172.21.15.159:6789/0 conn(0x7f32d40b4d60 legacy=0x7f32d413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:38.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 >> v1:172.21.15.134:6789/0 conn(0x7f32d414be90 legacy=0x7f32d414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 1 monclient: found mon.noname-b 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d40b1030 con 0x7f32d40b1730 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32da7f4640 5 monclient: authenticate success, global_id 34190 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32da7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:38.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f32c8001650 con 0x7f32d40b1730 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32d2ffd640 10 monclient: dump: 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:38.120 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:38.121 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:38.121 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.117+0000 7f32da7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:38.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32c80018a0 con 0x7f32d40b1730 2024-03-25T17:29:38.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32d2ffd640 1 -- 172.21.15.134:0/3843431007 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f32c8001c30 con 0x7f32d40b1730 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 4 set_mon_vals no callback set 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32d2ffd640 10 monclient: dump: 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.124 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 10 monclient: shutdown 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 1 -- 172.21.15.134:0/3843431007 >> v1:172.21.15.134:6790/0 conn(0x7f32d40b1730 legacy=0x7f32d40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 1 -- 172.21.15.134:0/3843431007 shutdown_connections 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 1 -- 172.21.15.134:0/3843431007 >> 172.21.15.134:0/3843431007 conn(0x7f32d40617a0 msgr2=0x7f32d4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 1 -- 172.21.15.134:0/3843431007 shutdown_connections 2024-03-25T17:29:38.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.119+0000 7f32da7f4640 1 -- 172.21.15.134:0/3843431007 wait complete. 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.120+0000 7f32da7f4640 10 monclient: build_initial_monmap 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.120+0000 7f32da7f4640 10 monclient: monmap: 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:38.126 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.120+0000 7f32da7f4640 1 librados: starting msgr at 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.120+0000 7f32da7f4640 1 librados: starting objecter 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 1 Processor -- start 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 1 -- start start 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 1 librados: setting wanted keys 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 1 librados: calling monclient init 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient: init 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:38.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f32d40b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f32d40b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f32d414be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d41381d0 con 0x7f32d414be90 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d41d78d0 con 0x7f32d40b4d60 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d4143900 con 0x7f32d40b1730 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.121+0000 7f32da7f4640 10 monclient(hunting): authenticate will time out at 5331.384655s 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d3fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f32d40b1730 0x7f32d4136810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:55422/0 (socket says 172.21.15.134:55422) 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d3fff640 1 -- 172.21.15.134:0/2387572412 learned_addr learned my addr 172.21.15.134:0/2387572412 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d8d91640 1 --1- 172.21.15.134:0/2387572412 >> v1:172.21.15.134:6789/0 conn(0x7f32d414be90 0x7f32d412fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:33936/0 (socket says 172.21.15.134:33936) 2024-03-25T17:29:38.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d0ff9640 10 client.?.objecter ms_handle_connect 0x7f32d40b1730 2024-03-25T17:29:38.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:38.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d0ff9640 10 client.?.objecter ms_handle_connect 0x7f32d414be90 2024-03-25T17:29:38.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:38.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d0ff9640 10 client.?.objecter ms_handle_connect 0x7f32d40b4d60 2024-03-25T17:29:38.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.122+0000 7f32d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:38.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677375042 0 0) 0x7f32d41d78d0 con 0x7f32d40b4d60 2024-03-25T17:29:38.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:38.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:38.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:38.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): my global_id is 4213 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32c0004eb0 con 0x7f32d40b4d60 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 78599565 0 0) 0x7f32d4143900 con 0x7f32d40b1730 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): my global_id is 34193 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d41d78d0 con 0x7f32d40b1730 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516755022 0 0) 0x7f32d41381d0 con 0x7f32d414be90 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:38.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 10 monclient(hunting): my global_id is 4200 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.124+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d4143900 con 0x7f32d414be90 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.128+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183242506 0 0) 0x7f32c0004eb0 con 0x7f32d40b4d60 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32d41381d0 con 0x7f32d40b4d60 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f32c8001bb0 con 0x7f32d40b4d60 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.133 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 882617575 0 0) 0x7f32d41d78d0 con 0x7f32d40b1730 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32c0004eb0 con 0x7f32d40b1730 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f32c4001970 con 0x7f32d40b1730 2024-03-25T17:29:38.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471161549 0 0) 0x7f32d4143900 con 0x7f32d414be90 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32d41d78d0 con 0x7f32d414be90 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f32cc001b50 con 0x7f32d414be90 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.129+0000 7f32d0ff9640 10 monclient(hunting): dump: 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:38.136 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:38.137 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:38.137 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:38.137 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2988486679 0 0) 0x7f32c0004eb0 con 0x7f32d40b1730 2024-03-25T17:29:38.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 >> v1:172.21.15.159:6789/0 conn(0x7f32d40b4d60 legacy=0x7f32d41d7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:38.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 >> v1:172.21.15.134:6789/0 conn(0x7f32d414be90 legacy=0x7f32d412fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:38.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 1 monclient: found mon.c 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d4143a90 con 0x7f32d40b1730 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 10 monclient: _finish_auth 0 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 5 monclient: authenticate success, global_id 34193 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 10 monclient: _renew_subs 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d4136fe0 con 0x7f32d40b1730 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 10 client.34193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:38.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 10 monclient: _renew_subs 2024-03-25T17:29:38.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:38.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d41d86a0 con 0x7f32d40b1730 2024-03-25T17:29:38.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.132+0000 7f32da7f4640 1 librados: init done 2024-03-25T17:29:38.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.133+0000 7f32da7f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:38.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.133+0000 7f32da7f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:38.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.133+0000 7f32da7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:38.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.133+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d40afbd0 con 0x7f32d40b1730 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.147+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32c4003610 con 0x7f32d40b1730 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.147+0000 7f32d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.147+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f32c4003800 con 0x7f32d40b1730 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.147+0000 7f32d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.147+0000 7f32d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.147+0000 7f32d0ff9640 10 monclient: dump: 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:38.148 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:38.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.147+0000 7f32d27fc640 4 set_mon_vals no callback set 2024-03-25T17:29:38.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.148+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f32c4003a20 con 0x7f32d40b1730 2024-03-25T17:29:38.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.148+0000 7f32d37fe640 10 monclient: get_auth_request con 0x7f32c006e0b0 auth_method 0 2024-03-25T17:29:38.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.149+0000 7f32d0ff9640 10 client.34193.objecter ms_handle_connect 0x7f32c006e0b0 2024-03-25T17:29:38.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.150+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f32c4054580 con 0x7f32d40b1730 2024-03-25T17:29:38.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.150+0000 7f32d0ff9640 10 client.34193.objecter ms_dispatch 0x7f32d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:38.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.150+0000 7f32d0ff9640 3 client.34193.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:38.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.150+0000 7f32d0ff9640 3 client.34193.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:38.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.150+0000 7f32d0ff9640 20 client.34193.objecter dump_active .. 0 homeless 2024-03-25T17:29:38.160 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:09.542 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.203 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.590 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.216 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.621 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.260 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:09.678 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.297 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.679 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.334 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:09.736 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.338 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.713 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.346 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:09.751 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.391+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32c4050090 con 0x7f32d40b1730 2024-03-25T17:29:38.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.391+0000 7f32d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:38.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.391+0000 7f32d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:38.448 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:09.830 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.471 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:09.858 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.506 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:09.911 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.550 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:09.968 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.565+0000 7f32da7f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:38.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.565+0000 7f32da7f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:38.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.565+0000 7f32da7f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:29:38.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.565+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32d40627b0 con 0x7f32d40b1730 2024-03-25T17:29:38.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.578+0000 7f32d0ff9640 1 -- 172.21.15.134:0/2387572412 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f32c4051fc0 con 0x7f32d40b1730 2024-03-25T17:29:38.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.578+0000 7f32d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:38.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.578+0000 7f32d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:38.579 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:29:38.580 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:38.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 10 librados: watch_flush enter 2024-03-25T17:29:38.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 10 librados: watch_flush exit 2024-03-25T17:29:38.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 20 client.34193.objecter shutdown clearing up homeless session... 2024-03-25T17:29:38.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 10 client.34193.objecter successfully canceled tick 2024-03-25T17:29:38.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 >> v1:172.21.15.134:6800/610837766 conn(0x7f32c006e0b0 legacy=0x7f32c0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:38.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 10 monclient: shutdown 2024-03-25T17:29:38.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:38.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.580+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 >> v1:172.21.15.134:6790/0 conn(0x7f32d40b1730 legacy=0x7f32d4136810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:38.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.581+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 shutdown_connections 2024-03-25T17:29:38.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.581+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 >> 172.21.15.134:0/2387572412 conn(0x7f32d40617a0 msgr2=0x7f32d40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:38.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.581+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 shutdown_connections 2024-03-25T17:29:38.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.581+0000 7f32da7f4640 1 -- 172.21.15.134:0/2387572412 wait complete. 2024-03-25T17:29:38.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:38.581+0000 7f32da7f4640 1 librados: shutdown 2024-03-25T17:29:38.587 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:09.969 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.591 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:38.604 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:09.979 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.610 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:10.012 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.621 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:10.026 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.740 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:10.127 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.744 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:10.127 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.800 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:10.205 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.841 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:10.259 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.881 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:10.256 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.884 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:10.267 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.885 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:10.287 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:38.895 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:10.300 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.058 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:10.441 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.091 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:10.496 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.140 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:10.558 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.154 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:10.556 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.163 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:10.567 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.172 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:10.555 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.216 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:10.603 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.270 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:10.657 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.353 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:10.728 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.382 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:10.787 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.408 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:10.783 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.436 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:10.838 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.444 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:10.862 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.600 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:10.983 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.636 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:11.040 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.655 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:11.038 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.688 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:11.093 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.730 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:11.113 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.785 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:11.167 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.931 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:11.333 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.953 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:11.357 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:39.985 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:11.387 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:40.016 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:11.420 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:40.016 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:11.433 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:40.087 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:11.505 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:44.746 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:16.150 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:44.764 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:16.152 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:45.095 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:16.470 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:45.221 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:16.604 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:45.397 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:45.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.489+0000 7ff257686640 10 monclient: get_monmap_and_config 2024-03-25T17:29:45.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.489+0000 7ff257686640 10 monclient: build_initial_monmap 2024-03-25T17:29:45.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.489+0000 7ff257686640 10 monclient: monmap: 2024-03-25T17:29:45.490 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:45.490 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:45.490 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:29:45.490539+0000 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:29:45.490539+0000 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.491 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 1 Processor -- start 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 1 -- start start 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 10 monclient: init 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 10 monclient(hunting): picked mon.noname-a con 0x7ff2500b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 10 monclient(hunting): picked mon.noname-c con 0x7ff2500b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.490+0000 7ff257686640 10 monclient(hunting): picked mon.noname-b con 0x7ff25014be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.491+0000 7ff257686640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2500af940 con 0x7ff2500b4d60 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.491+0000 7ff257686640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2500b0a70 con 0x7ff2500b1730 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.491+0000 7ff257686640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2500b0d50 con 0x7ff25014be90 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.491+0000 7ff257686640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:45.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.491+0000 7ff257686640 10 monclient(hunting): authenticate will time out at 5338.753921s 2024-03-25T17:29:45.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.491+0000 7ff255c23640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7ff25014be90 0x7ff25014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:36800/0 (socket says 172.21.15.134:36800) 2024-03-25T17:29:45.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.491+0000 7ff255c23640 1 -- 172.21.15.134:0/3724622356 learned_addr learned my addr 172.21.15.134:0/3724622356 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552432905 0 0) 0x7ff2500b0a70 con 0x7ff2500b1730 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): my global_id is 4216 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff23c003520 con 0x7ff2500b1730 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658129330 0 0) 0x7ff2500b0d50 con 0x7ff25014be90 2024-03-25T17:29:45.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:45.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:45.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:45.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 10 monclient(hunting): my global_id is 34196 2024-03-25T17:29:45.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.494+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2500b0a70 con 0x7ff25014be90 2024-03-25T17:29:45.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.496+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374107454 0 0) 0x7ff2500af940 con 0x7ff2500b4d60 2024-03-25T17:29:45.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.496+0000 7ff247fff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:45.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.496+0000 7ff247fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:45.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.496+0000 7ff247fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:45.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.496+0000 7ff247fff640 10 monclient(hunting): my global_id is 4203 2024-03-25T17:29:45.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.496+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2500b0d50 con 0x7ff2500b4d60 2024-03-25T17:29:45.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2690116100 0 0) 0x7ff23c003520 con 0x7ff2500b1730 2024-03-25T17:29:45.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 >> v1:172.21.15.134:6790/0 conn(0x7ff25014be90 legacy=0x7ff25014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:45.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 >> v1:172.21.15.134:6789/0 conn(0x7ff2500b4d60 legacy=0x7ff25013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:45.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:45.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 1 monclient: found mon.noname-c 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2500b1030 con 0x7ff2500b1730 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 10 monclient: _finish_auth 0 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff240001690 con 0x7ff2500b1730 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff247fff640 10 monclient: dump: 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.500 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:45.501 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:45.501 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:45.501 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:45.501 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:45.501 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff257686640 5 monclient: authenticate success, global_id 4216 2024-03-25T17:29:45.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.498+0000 7ff257686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:45.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2400017f0 con 0x7ff2500b1730 2024-03-25T17:29:45.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff247fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:45.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff247fff640 1 -- 172.21.15.134:0/3724622356 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff240001c00 con 0x7ff2500b1730 2024-03-25T17:29:45.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff247fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff247fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff247fff640 10 monclient: dump: 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff257686640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.500+0000 7ff257686640 4 set_mon_vals no callback set 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.501+0000 7ff257686640 10 monclient: shutdown 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.501+0000 7ff257686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.501+0000 7ff257686640 1 -- 172.21.15.134:0/3724622356 >> v1:172.21.15.159:6789/0 conn(0x7ff2500b1730 legacy=0x7ff2500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.502+0000 7ff257686640 1 -- 172.21.15.134:0/3724622356 shutdown_connections 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.502+0000 7ff257686640 1 -- 172.21.15.134:0/3724622356 >> 172.21.15.134:0/3724622356 conn(0x7ff2500617a0 msgr2=0x7ff250061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.502+0000 7ff257686640 1 -- 172.21.15.134:0/3724622356 shutdown_connections 2024-03-25T17:29:45.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.502+0000 7ff257686640 1 -- 172.21.15.134:0/3724622356 wait complete. 2024-03-25T17:29:45.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.503+0000 7ff257686640 10 monclient: build_initial_monmap 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.503+0000 7ff257686640 10 monclient: monmap: 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.503+0000 7ff257686640 1 librados: starting msgr at 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.503+0000 7ff257686640 1 librados: starting objecter 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.503+0000 7ff257686640 1 Processor -- start 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 1 -- start start 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 1 librados: setting wanted keys 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 1 librados: calling monclient init 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 10 monclient: init 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 10 monclient(hunting): picked mon.noname-b con 0x7ff2500b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.504+0000 7ff257686640 10 monclient(hunting): picked mon.noname-a con 0x7ff2500b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff257686640 10 monclient(hunting): picked mon.noname-c con 0x7ff25014be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff257686640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25013e960 con 0x7ff2500b4d60 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff257686640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff250141b30 con 0x7ff2500b1730 2024-03-25T17:29:45.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff257686640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff250137b80 con 0x7ff25014be90 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff257686640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff257686640 10 monclient(hunting): authenticate will time out at 5338.767923s 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff255c23640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7ff25014be90 0x7ff2501e2f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:36814/0 (socket says 172.21.15.134:36814) 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff255c23640 1 -- 172.21.15.134:0/725382318 learned_addr learned my addr 172.21.15.134:0/725382318 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff245ffb640 10 client.?.objecter ms_handle_connect 0x7ff25014be90 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff245ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.505+0000 7ff245ffb640 10 client.?.objecter ms_handle_connect 0x7ff2500b1730 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.506+0000 7ff245ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.506+0000 7ff245ffb640 10 client.?.objecter ms_handle_connect 0x7ff2500b4d60 2024-03-25T17:29:45.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.506+0000 7ff245ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438316527 0 0) 0x7ff250137b80 con 0x7ff25014be90 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): my global_id is 34199 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff234004eb0 con 0x7ff25014be90 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960230346 0 0) 0x7ff250141b30 con 0x7ff2500b1730 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:45.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 10 monclient(hunting): my global_id is 4219 2024-03-25T17:29:45.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.508+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff250137b80 con 0x7ff2500b1730 2024-03-25T17:29:45.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.509+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494608073 0 0) 0x7ff25013e960 con 0x7ff2500b4d60 2024-03-25T17:29:45.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.509+0000 7ff245ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:29:45.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.509+0000 7ff245ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:45.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.509+0000 7ff245ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:45.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.509+0000 7ff245ffb640 10 monclient(hunting): my global_id is 4206 2024-03-25T17:29:45.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.509+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff250141b30 con 0x7ff2500b4d60 2024-03-25T17:29:45.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.513+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1437666594 0 0) 0x7ff250137b80 con 0x7ff2500b1730 2024-03-25T17:29:45.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.513+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff25013e960 con 0x7ff2500b1730 2024-03-25T17:29:45.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.513+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff24c001870 con 0x7ff2500b1730 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.513+0000 7ff245ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.513+0000 7ff245ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.513+0000 7ff245ffb640 10 monclient(hunting): dump: 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.513+0000 7ff245ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:45.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.514+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3022994234 0 0) 0x7ff234004eb0 con 0x7ff25014be90 2024-03-25T17:29:45.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.514+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff250137b80 con 0x7ff25014be90 2024-03-25T17:29:45.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.514+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff248001c70 con 0x7ff25014be90 2024-03-25T17:29:45.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.514+0000 7ff245ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:45.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.514+0000 7ff245ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.514+0000 7ff245ffb640 10 monclient(hunting): dump: 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.516+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3815346205 0 0) 0x7ff250141b30 con 0x7ff2500b4d60 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.516+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff234004eb0 con 0x7ff2500b4d60 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.516+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff240001b00 con 0x7ff2500b4d60 2024-03-25T17:29:45.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.516+0000 7ff245ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.516+0000 7ff245ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.516+0000 7ff245ffb640 10 monclient(hunting): dump: 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3372429201 0 0) 0x7ff25013e960 con 0x7ff2500b1730 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 >> v1:172.21.15.134:6790/0 conn(0x7ff25014be90 legacy=0x7ff2501e2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:45.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 >> v1:172.21.15.134:6789/0 conn(0x7ff2500b4d60 legacy=0x7ff2501dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 1 monclient: found mon.b 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250137d10 con 0x7ff2500b1730 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 10 monclient: _finish_auth 0 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff245ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 5 monclient: authenticate success, global_id 4219 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 10 monclient: _renew_subs 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2501386c0 con 0x7ff2500b1730 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:45.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 10 monclient: _renew_subs 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2501356a0 con 0x7ff2500b1730 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.517+0000 7ff257686640 1 librados: init done 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.519+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff24c003610 con 0x7ff2500b1730 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.519+0000 7ff245ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.519+0000 7ff2477fe640 4 set_mon_vals no callback set 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.519+0000 7ff23b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.519+0000 7ff23b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.519+0000 7ff23b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:45.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.519+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff230005080 con 0x7ff2500b1730 2024-03-25T17:29:45.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.532+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff24c001d80 con 0x7ff2500b1730 2024-03-25T17:29:45.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.532+0000 7ff245ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:45.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.532+0000 7ff245ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:45.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.532+0000 7ff245ffb640 10 monclient: dump: 2024-03-25T17:29:45.533 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:45.533 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:45.534 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:45.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.533+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7ff24c0197d0 con 0x7ff2500b1730 2024-03-25T17:29:45.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.534+0000 7ff254c21640 10 monclient: get_auth_request con 0x7ff23406e0f0 auth_method 0 2024-03-25T17:29:45.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.534+0000 7ff245ffb640 10 client.4219.objecter ms_handle_connect 0x7ff23406e0f0 2024-03-25T17:29:45.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.535+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7ff24c0535c0 con 0x7ff2500b1730 2024-03-25T17:29:45.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.535+0000 7ff245ffb640 10 client.4219.objecter ms_dispatch 0x7ff250000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:45.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.535+0000 7ff245ffb640 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:45.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.535+0000 7ff245ffb640 3 client.4219.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:45.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.535+0000 7ff245ffb640 20 client.4219.objecter dump_active .. 0 homeless 2024-03-25T17:29:45.568 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:16.950 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:45.643 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:17.047 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:45.796 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:17.199 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:45.842 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:17.260 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:45.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.880+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff24c04ef00 con 0x7ff2500b1730 2024-03-25T17:29:45.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.880+0000 7ff245ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:45.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:45.880+0000 7ff245ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:46.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.101+0000 7ff23b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:46.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.102+0000 7ff23b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:46.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.102+0000 7ff23b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:46.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.102+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff230005330 con 0x7ff2500b1730 2024-03-25T17:29:46.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.117+0000 7ff245ffb640 1 -- 172.21.15.134:0/725382318 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7ff24c051000 con 0x7ff2500b1730 2024-03-25T17:29:46.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.117+0000 7ff245ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:46.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.117+0000 7ff245ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:46.119 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:29:46.119 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:46.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 10 librados: watch_flush enter 2024-03-25T17:29:46.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 10 librados: watch_flush exit 2024-03-25T17:29:46.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 20 client.4219.objecter shutdown clearing up homeless session... 2024-03-25T17:29:46.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 10 client.4219.objecter successfully canceled tick 2024-03-25T17:29:46.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 >> v1:172.21.15.134:6800/610837766 conn(0x7ff23406e0f0 legacy=0x7ff2340705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:46.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 10 monclient: shutdown 2024-03-25T17:29:46.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:46.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 >> v1:172.21.15.159:6789/0 conn(0x7ff2500b1730 legacy=0x7ff250134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:46.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 shutdown_connections 2024-03-25T17:29:46.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 >> 172.21.15.134:0/725382318 conn(0x7ff2500617a0 msgr2=0x7ff2500b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:46.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 shutdown_connections 2024-03-25T17:29:46.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 1 -- 172.21.15.134:0/725382318 wait complete. 2024-03-25T17:29:46.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:46.120+0000 7ff23b7fe640 1 librados: shutdown 2024-03-25T17:29:46.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:50.730 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:22.134 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:50.734 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:22.138 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:50.835 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:22.222 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:50.839 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:22.226 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:50.951 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:22.326 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:50.955 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:22.330 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.258 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:22.641 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.262 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:22.644 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.338 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:22.721 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.344 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:22.727 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.414 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:22.818 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.418 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:22.823 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.743 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:23.145 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:51.749 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:23.151 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:52.047 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:23.464 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:52.053 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:23.470 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:52.938 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.042+0000 7f306de0d640 10 monclient: get_monmap_and_config 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.042+0000 7f306de0d640 10 monclient: build_initial_monmap 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.042+0000 7f306de0d640 10 monclient: monmap: 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:29:53.043669+0000 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:29:53.043669+0000 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:53.043 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:53.044 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:29:53.044 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:29:53.044 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:53.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.043+0000 7f306de0d640 1 Processor -- start 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.043+0000 7f306de0d640 1 -- start start 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.043+0000 7f306de0d640 10 monclient: init 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 10 monclient(hunting): picked mon.noname-c con 0x7f30680b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 10 monclient(hunting): picked mon.noname-b con 0x7f30680b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 10 monclient(hunting): picked mon.noname-a con 0x7f306814be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680af940 con 0x7f306814be90 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680b0a70 con 0x7f30680b4d60 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680b0d50 con 0x7f30680b1730 2024-03-25T17:29:53.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f306de0d640 10 monclient(hunting): authenticate will time out at 5346.307405s 2024-03-25T17:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f3066ffd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f30680b1730 0x7f30680b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:40116/0 (socket says 172.21.15.134:40116) 2024-03-25T17:29:53.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.044+0000 7f3066ffd640 1 -- 172.21.15.134:0/2518914990 learned_addr learned my addr 172.21.15.134:0/2518914990 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004979427 0 0) 0x7f30680b0d50 con 0x7f30680b1730 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): my global_id is 34202 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3050003520 con 0x7f30680b1730 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138606787 0 0) 0x7f30680b0a70 con 0x7f30680b4d60 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:53.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 10 monclient(hunting): my global_id is 4222 2024-03-25T17:29:53.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.047+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30680b0d50 con 0x7f30680b4d60 2024-03-25T17:29:53.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.049+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362157648 0 0) 0x7f30680af940 con 0x7f306814be90 2024-03-25T17:29:53.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.049+0000 7f30667fc640 10 monclient(hunting): _negotiate 2024-03-25T17:29:53.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.049+0000 7f30667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:53.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.049+0000 7f30667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:53.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.049+0000 7f30667fc640 10 monclient(hunting): my global_id is 4209 2024-03-25T17:29:53.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.049+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30680b0a70 con 0x7f306814be90 2024-03-25T17:29:53.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057555675 0 0) 0x7f3050003520 con 0x7f30680b1730 2024-03-25T17:29:53.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 >> v1:172.21.15.159:6789/0 conn(0x7f30680b4d60 legacy=0x7f306813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:53.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 >> v1:172.21.15.134:6789/0 conn(0x7f306814be90 legacy=0x7f306814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:53.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 10 monclient: _finish_hunting 0 2024-03-25T17:29:53.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 1 monclient: found mon.noname-b 2024-03-25T17:29:53.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680b1030 con 0x7f30680b1730 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 10 monclient: _finish_auth 0 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 10 monclient: _check_auth_tickets 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f305c001690 con 0x7f30680b1730 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f30667fc640 10 monclient: dump: 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:53.055 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f306de0d640 5 monclient: authenticate success, global_id 34202 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.051+0000 7f306de0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f305c0018e0 con 0x7f30680b1730 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f30667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f30667fc640 1 -- 172.21.15.134:0/2518914990 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f305c001c70 con 0x7f30680b1730 2024-03-25T17:29:53.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f30667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f30667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f30667fc640 10 monclient: dump: 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:53.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f306de0d640 10 monclient: get_monmap_and_config success 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.052+0000 7f306de0d640 4 set_mon_vals no callback set 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.053+0000 7f306de0d640 10 monclient: shutdown 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.053+0000 7f306de0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.053+0000 7f306de0d640 1 -- 172.21.15.134:0/2518914990 >> v1:172.21.15.134:6790/0 conn(0x7f30680b1730 legacy=0x7f30680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.053+0000 7f306de0d640 1 -- 172.21.15.134:0/2518914990 shutdown_connections 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.053+0000 7f306de0d640 1 -- 172.21.15.134:0/2518914990 >> 172.21.15.134:0/2518914990 conn(0x7f30680617a0 msgr2=0x7f3068061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.053+0000 7f306de0d640 1 -- 172.21.15.134:0/2518914990 shutdown_connections 2024-03-25T17:29:53.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.053+0000 7f306de0d640 1 -- 172.21.15.134:0/2518914990 wait complete. 2024-03-25T17:29:53.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.054+0000 7f306de0d640 10 monclient: build_initial_monmap 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.054+0000 7f306de0d640 10 monclient: monmap: 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:53.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.054+0000 7f306de0d640 1 librados: starting msgr at 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.054+0000 7f306de0d640 1 librados: starting objecter 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 1 Processor -- start 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 1 -- start start 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 1 librados: setting wanted keys 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 1 librados: calling monclient init 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 10 monclient: init 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 10 monclient: _reopen_session rank -1 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 10 monclient(hunting): picked mon.noname-c con 0x7f30680b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:29:53.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.055+0000 7f306de0d640 10 monclient(hunting): picked mon.noname-a con 0x7f30680b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f306de0d640 10 monclient(hunting): picked mon.noname-b con 0x7f306814be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f306de0d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3068141140 con 0x7f30680b4d60 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f306de0d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30681e8e10 con 0x7f306814be90 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f306de0d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3068141390 con 0x7f30680b1730 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f306de0d640 10 monclient(hunting): _renew_subs 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f306de0d640 10 monclient(hunting): authenticate will time out at 5346.318962s 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f30677fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f30680b1730 0x7f3068138080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:40126/0 (socket says 172.21.15.134:40126) 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f30677fe640 1 -- 172.21.15.134:0/3341498134 learned_addr learned my addr 172.21.15.134:0/3341498134 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f304ffff640 10 client.?.objecter ms_handle_connect 0x7f30680b1730 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f304ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f304ffff640 10 client.?.objecter ms_handle_connect 0x7f306814be90 2024-03-25T17:29:53.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.056+0000 7f304ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478693057 0 0) 0x7f3068141390 con 0x7f30680b1730 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): my global_id is 34205 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3054004eb0 con 0x7f30680b1730 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 client.?.objecter ms_handle_connect 0x7f30680b4d60 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195611299 0 0) 0x7f30681e8e10 con 0x7f306814be90 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:53.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.058+0000 7f304ffff640 10 monclient(hunting): my global_id is 4225 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.059+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3068141390 con 0x7f306814be90 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.062+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522656296 0 0) 0x7f3068141140 con 0x7f30680b4d60 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.062+0000 7f304ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.062+0000 7f304ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.062+0000 7f304ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.062+0000 7f304ffff640 10 monclient(hunting): my global_id is 4212 2024-03-25T17:29:53.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.062+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30681e8e10 con 0x7f30680b4d60 2024-03-25T17:29:53.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.063+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3362075155 0 0) 0x7f3068141390 con 0x7f306814be90 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.063+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3068141140 con 0x7f306814be90 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.063+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3060001bd0 con 0x7f306814be90 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.063+0000 7f304ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.063+0000 7f304ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.063+0000 7f304ffff640 10 monclient(hunting): dump: 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:53.069 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.063+0000 7f304ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.065+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 781034896 0 0) 0x7f3054004eb0 con 0x7f30680b1730 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.065+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3068141390 con 0x7f30680b1730 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4003624827 0 0) 0x7f3068141140 con 0x7f306814be90 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 >> v1:172.21.15.134:6790/0 conn(0x7f30680b1730 legacy=0x7f3068138080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 >> v1:172.21.15.134:6789/0 conn(0x7f30680b4d60 legacy=0x7f30681e7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 1 monclient: found mon.b 2024-03-25T17:29:53.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068141670 con 0x7f306814be90 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 10 monclient: _finish_auth 0 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f304ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 5 monclient: authenticate success, global_id 4225 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 10 monclient: _renew_subs 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30681435d0 con 0x7f306814be90 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 10 monclient: _renew_subs 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3068141c70 con 0x7f306814be90 2024-03-25T17:29:53.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.068+0000 7f306de0d640 1 librados: init done 2024-03-25T17:29:53.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.069+0000 7f306de0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:53.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.069+0000 7f306de0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:53.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.069+0000 7f306de0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:53.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.069+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3068140540 con 0x7f306814be90 2024-03-25T17:29:53.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.086+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3060002080 con 0x7f306814be90 2024-03-25T17:29:53.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.086+0000 7f304ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.086+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3060003b30 con 0x7f306814be90 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.086+0000 7f304ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.086+0000 7f304ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.086+0000 7f304ffff640 10 monclient: dump: 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:29:53.087 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:29:53.088 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:29:53.088 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:29:53.088 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:29:53.088 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:29:53.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.086+0000 7f3065ffb640 4 set_mon_vals no callback set 2024-03-25T17:29:53.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.087+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f3060019b40 con 0x7f306814be90 2024-03-25T17:29:53.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.088+0000 7f30677fe640 10 monclient: get_auth_request con 0x7f305406e040 auth_method 0 2024-03-25T17:29:53.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.088+0000 7f304ffff640 10 client.4225.objecter ms_handle_connect 0x7f305406e040 2024-03-25T17:29:53.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.088+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f3060053770 con 0x7f306814be90 2024-03-25T17:29:53.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.088+0000 7f304ffff640 10 client.4225.objecter ms_dispatch 0x7f3068000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:29:53.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.088+0000 7f304ffff640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:29:53.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.088+0000 7f304ffff640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:29:53.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.088+0000 7f304ffff640 20 client.4225.objecter dump_active .. 0 homeless 2024-03-25T17:29:53.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.348+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f306004f0b0 con 0x7f306814be90 2024-03-25T17:29:53.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.348+0000 7f304ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:29:53.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.348+0000 7f304ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:29:53.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.688+0000 7f306de0d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:53.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.688+0000 7f306de0d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:53.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.688+0000 7f306de0d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:29:53.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.688+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30680627b0 con 0x7f306814be90 2024-03-25T17:29:53.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.714+0000 7f304ffff640 1 -- 172.21.15.134:0/3341498134 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f30600511b0 con 0x7f306814be90 2024-03-25T17:29:53.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.714+0000 7f304ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:29:53.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.714+0000 7f304ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:29:53.715 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:29:53.715 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:29:53.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.716+0000 7f306de0d640 10 librados: watch_flush enter 2024-03-25T17:29:53.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.716+0000 7f306de0d640 10 librados: watch_flush exit 2024-03-25T17:29:53.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 20 client.4225.objecter shutdown clearing up homeless session... 2024-03-25T17:29:53.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 10 client.4225.objecter successfully canceled tick 2024-03-25T17:29:53.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 >> v1:172.21.15.134:6800/610837766 conn(0x7f305406e040 legacy=0x7f3054070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:53.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 10 monclient: shutdown 2024-03-25T17:29:53.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:29:53.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 >> v1:172.21.15.159:6789/0 conn(0x7f306814be90 legacy=0x7f30681e8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:29:53.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 shutdown_connections 2024-03-25T17:29:53.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 >> 172.21.15.134:0/3341498134 conn(0x7f30680617a0 msgr2=0x7f306813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:29:53.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 shutdown_connections 2024-03-25T17:29:53.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 1 -- 172.21.15.134:0/3341498134 wait complete. 2024-03-25T17:29:53.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:29:53.717+0000 7f306de0d640 1 librados: shutdown 2024-03-25T17:29:53.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:29:54.797 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:00:26.202 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:55.305 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:00:26.692 35440== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:55.394 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:00:26.776 35441== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:55.514 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:00:26.890 35443== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:55.856 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:00:27.261 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:56.040 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:00:27.422 35442== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:56.222 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:00:27.624 35537== Warning: unimplemented fcntl command: 1038 2024-03-25T17:29:56.712 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:00:28.129 35528== Warning: unimplemented fcntl command: 1038 2024-03-25T17:30:00.537 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.657+0000 7ffa08165640 10 monclient: get_monmap_and_config 2024-03-25T17:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.657+0000 7ffa08165640 10 monclient: build_initial_monmap 2024-03-25T17:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.657+0000 7ffa08165640 10 monclient: monmap: 2024-03-25T17:30:00.658 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:00.658962+0000 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:00.658962+0000 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:00.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.659+0000 7ffa08165640 1 Processor -- start 2024-03-25T17:30:00.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.659+0000 7ffa08165640 1 -- start start 2024-03-25T17:30:00.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.659+0000 7ffa08165640 10 monclient: init 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 10 monclient(hunting): picked mon.noname-b con 0x7ffa000a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 10 monclient(hunting): picked mon.noname-a con 0x7ffa0013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 10 monclient(hunting): picked mon.noname-c con 0x7ffa001430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa00142520 con 0x7ffa0013f710 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa000b28a0 con 0x7ffa001430c0 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa000b2a30 con 0x7ffa000a98f0 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.660+0000 7ffa08165640 10 monclient(hunting): authenticate will time out at 5353.923755s 2024-03-25T17:30:00.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.661+0000 7ffa05700640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ffa0013f710 0x7ffa001429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:51138/0 (socket says 172.21.15.134:51138) 2024-03-25T17:30:00.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.661+0000 7ffa05700640 1 -- 172.21.15.134:0/565607601 learned_addr learned my addr 172.21.15.134:0/565607601 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:00.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.665+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983419903 0 0) 0x7ffa00142520 con 0x7ffa0013f710 2024-03-25T17:30:00.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.665+0000 7ffa04eff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:00.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.665+0000 7ffa04eff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:00.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.665+0000 7ffa04eff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:00.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.665+0000 7ffa04eff640 10 monclient(hunting): my global_id is 4215 2024-03-25T17:30:00.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.665+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9ec003520 con 0x7ffa0013f710 2024-03-25T17:30:00.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 265664421 0 0) 0x7ff9ec003520 con 0x7ffa0013f710 2024-03-25T17:30:00.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 >> v1:172.21.15.134:6790/0 conn(0x7ffa000a98f0 legacy=0x7ffa000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 >> v1:172.21.15.159:6789/0 conn(0x7ffa001430c0 legacy=0x7ffa0014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 10 monclient: _finish_hunting 0 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 1 monclient: found mon.noname-a 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa000b44c0 con 0x7ffa0013f710 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 10 monclient: _finish_auth 0 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 10 monclient: _check_auth_tickets 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff9f0001690 con 0x7ffa0013f710 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa04eff640 10 monclient: dump: 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:00.672 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa08165640 5 monclient: authenticate success, global_id 4215 2024-03-25T17:30:00.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.671+0000 7ffa08165640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:00.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.673+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9f0001860 con 0x7ffa0013f710 2024-03-25T17:30:00.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.673+0000 7ffa04eff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:00.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.673+0000 7ffa08165640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:00.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.673+0000 7ffa08165640 4 set_mon_vals no callback set 2024-03-25T17:30:00.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa08165640 10 monclient: shutdown 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa04eff640 1 -- 172.21.15.134:0/565607601 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff9f0001cb0 con 0x7ffa0013f710 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa04eff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa04eff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa04eff640 10 monclient: dump: 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:00.676 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:00.677 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:00.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa08165640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:00.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa08165640 1 -- 172.21.15.134:0/565607601 >> v1:172.21.15.134:6789/0 conn(0x7ffa0013f710 legacy=0x7ffa001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:00.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa08165640 1 -- 172.21.15.134:0/565607601 shutdown_connections 2024-03-25T17:30:00.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.674+0000 7ffa08165640 1 -- 172.21.15.134:0/565607601 >> 172.21.15.134:0/565607601 conn(0x7ffa000b7ea0 msgr2=0x7ffa000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:00.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.675+0000 7ffa08165640 1 -- 172.21.15.134:0/565607601 shutdown_connections 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.675+0000 7ffa08165640 1 -- 172.21.15.134:0/565607601 wait complete. 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.676+0000 7ffa08165640 10 monclient: build_initial_monmap 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.676+0000 7ffa08165640 10 monclient: monmap: 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:00.680 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.677+0000 7ffa08165640 1 librados: starting msgr at 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.677+0000 7ffa08165640 1 librados: starting objecter 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.677+0000 7ffa08165640 1 Processor -- start 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.677+0000 7ffa08165640 1 -- start start 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.677+0000 7ffa08165640 1 librados: setting wanted keys 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.677+0000 7ffa08165640 1 librados: calling monclient init 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.677+0000 7ffa08165640 10 monclient: init 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 10 monclient(hunting): picked mon.noname-c con 0x7ffa000a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 10 monclient(hunting): picked mon.noname-b con 0x7ffa0013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 10 monclient(hunting): picked mon.noname-a con 0x7ffa001430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:00.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa000b7be0 con 0x7ffa001430c0 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0013b770 con 0x7ffa0013f710 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa0013ba50 con 0x7ffa000a98f0 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.678+0000 7ffa08165640 10 monclient(hunting): authenticate will time out at 5353.941317s 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.679+0000 7ffa06702640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ffa001430c0 0x7ffa0012d260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:51142/0 (socket says 172.21.15.134:51142) 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.679+0000 7ffa06702640 1 -- 172.21.15.134:0/3938397990 learned_addr learned my addr 172.21.15.134:0/3938397990 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.679+0000 7ff9f67fc640 10 client.?.objecter ms_handle_connect 0x7ffa001430c0 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.679+0000 7ff9f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.679+0000 7ff9f67fc640 10 client.?.objecter ms_handle_connect 0x7ffa0013f710 2024-03-25T17:30:00.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.679+0000 7ff9f67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:00.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.682+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649934901 0 0) 0x7ffa000b7be0 con 0x7ffa001430c0 2024-03-25T17:30:00.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.682+0000 7ff9f67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:00.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.682+0000 7ff9f67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:00.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.682+0000 7ff9f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:00.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.682+0000 7ff9f67fc640 10 monclient(hunting): my global_id is 4218 2024-03-25T17:30:00.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.682+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9e4004eb0 con 0x7ffa001430c0 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.690+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3160651460 0 0) 0x7ff9e4004eb0 con 0x7ffa001430c0 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.690+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa000b7be0 con 0x7ffa001430c0 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.690+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff9f8001c90 con 0x7ffa001430c0 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.690+0000 7ff9f67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.690+0000 7ff9f67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.690+0000 7ff9f67fc640 10 monclient(hunting): dump: 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:00.691 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:00.692 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:00.692 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:00.692 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:00.692 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:00.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.690+0000 7ff9f67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:00.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2058057978 0 0) 0x7ffa000b7be0 con 0x7ffa001430c0 2024-03-25T17:30:00.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 >> v1:172.21.15.134:6790/0 conn(0x7ffa000a98f0 legacy=0x7ffa0013c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:00.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 >> v1:172.21.15.159:6789/0 conn(0x7ffa0013f710 legacy=0x7ffa0012cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:00.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 10 monclient: _finish_hunting 0 2024-03-25T17:30:00.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 1 monclient: found mon.a 2024-03-25T17:30:00.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa0012d970 con 0x7ffa001430c0 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 10 monclient: _finish_auth 0 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 10 monclient: _check_auth_tickets 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ff9f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 5 monclient: authenticate success, global_id 4218 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 10 monclient: _renew_subs 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa000b77e0 con 0x7ffa001430c0 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 10 monclient: _renew_subs 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:00.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa0013aa80 con 0x7ffa001430c0 2024-03-25T17:30:00.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.695+0000 7ffa08165640 1 librados: init done 2024-03-25T17:30:00.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.697+0000 7ffa08165640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:00.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.697+0000 7ffa08165640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:00.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.697+0000 7ffa08165640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:00.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.697+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa000b3ef0 con 0x7ffa001430c0 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.721+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9f8003990 con 0x7ffa001430c0 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.721+0000 7ff9f67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.721+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff9f8003d60 con 0x7ffa001430c0 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.721+0000 7ff9f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.721+0000 7ff9f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.721+0000 7ff9f67fc640 10 monclient: dump: 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:00.723 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:00.724 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:00.724 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:00.724 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:00.724 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:00.724 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:00.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.721+0000 7ff9f7fff640 4 set_mon_vals no callback set 2024-03-25T17:30:00.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.722+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7ff9f8003f80 con 0x7ffa001430c0 2024-03-25T17:30:00.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.723+0000 7ffa05f01640 10 monclient: get_auth_request con 0x7ff9e406dbe0 auth_method 0 2024-03-25T17:30:00.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.723+0000 7ff9f67fc640 10 client.4218.objecter ms_handle_connect 0x7ff9e406dbe0 2024-03-25T17:30:00.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.725+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7ff9f8054b70 con 0x7ffa001430c0 2024-03-25T17:30:00.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.725+0000 7ff9f67fc640 10 client.4218.objecter ms_dispatch 0x7ffa00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:00.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.725+0000 7ff9f67fc640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:00.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.725+0000 7ff9f67fc640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:00.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.725+0000 7ff9f67fc640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T17:30:00.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.987+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9f8050680 con 0x7ffa001430c0 2024-03-25T17:30:00.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.987+0000 7ff9f67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:00.988 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:00.987+0000 7ff9f67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:01.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.268+0000 7ffa08165640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:01.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.268+0000 7ffa08165640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:01.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.268+0000 7ffa08165640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:01.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.268+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa000627b0 con 0x7ffa001430c0 2024-03-25T17:30:01.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.336+0000 7ff9f67fc640 1 -- 172.21.15.134:0/3938397990 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7ff9f80525b0 con 0x7ffa001430c0 2024-03-25T17:30:01.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.336+0000 7ff9f67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:01.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.336+0000 7ff9f67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:01.337 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:01.337 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:01.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 10 librados: watch_flush enter 2024-03-25T17:30:01.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 10 librados: watch_flush exit 2024-03-25T17:30:01.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T17:30:01.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 10 client.4218.objecter successfully canceled tick 2024-03-25T17:30:01.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 >> v1:172.21.15.134:6800/610837766 conn(0x7ff9e406dbe0 legacy=0x7ff9e40700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:01.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 10 monclient: shutdown 2024-03-25T17:30:01.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:01.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 >> v1:172.21.15.134:6789/0 conn(0x7ffa001430c0 legacy=0x7ffa0012d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:01.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 shutdown_connections 2024-03-25T17:30:01.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.339+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 >> 172.21.15.134:0/3938397990 conn(0x7ffa000b7ea0 msgr2=0x7ffa00074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:01.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.340+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 shutdown_connections 2024-03-25T17:30:01.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.340+0000 7ffa08165640 1 -- 172.21.15.134:0/3938397990 wait complete. 2024-03-25T17:30:01.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:01.340+0000 7ffa08165640 1 librados: shutdown 2024-03-25T17:30:01.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:03.609 INFO:tasks.ceph.osd.0.smithi134.stderr:2024-03-25T17:30:03.605+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T17:30:03.712 INFO:tasks.ceph.osd.5.smithi159.stderr:2024-03-25T17:30:03.708+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T17:30:03.742 INFO:tasks.ceph.osd.7.smithi159.stderr:2024-03-25T17:30:03.738+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T17:30:03.749 INFO:tasks.ceph.osd.4.smithi159.stderr:2024-03-25T17:30:03.746+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T17:30:04.517 INFO:tasks.ceph.osd.6.smithi159.stderr:2024-03-25T17:30:04.507+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T17:30:04.932 INFO:tasks.ceph.osd.3.smithi134.stderr:2024-03-25T17:30:04.929+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T17:30:05.254 INFO:tasks.ceph.osd.2.smithi134.stderr:2024-03-25T17:30:05.250+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T17:30:05.793 INFO:tasks.ceph.osd.1.smithi134.stderr:2024-03-25T17:30:05.789+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T17:30:08.161 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:08.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.274+0000 7f5707923640 10 monclient: get_monmap_and_config 2024-03-25T17:30:08.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.274+0000 7f5707923640 10 monclient: build_initial_monmap 2024-03-25T17:30:08.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.274+0000 7f5707923640 10 monclient: monmap: 2024-03-25T17:30:08.275 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:08.275947+0000 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:08.275947+0000 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:08.276 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:08.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.275+0000 7f5707923640 1 Processor -- start 2024-03-25T17:30:08.277 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.275+0000 7f5707923640 1 -- start start 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.275+0000 7f5707923640 10 monclient: init 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 10 monclient(hunting): picked mon.noname-c con 0x7f57000a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 10 monclient(hunting): picked mon.noname-a con 0x7f570013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 10 monclient(hunting): picked mon.noname-b con 0x7f57001430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5700142520 con 0x7f570013f710 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57000b28a0 con 0x7f57000a98f0 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57000b2a30 con 0x7f57001430c0 2024-03-25T17:30:08.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:08.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.276+0000 7f5707923640 10 monclient(hunting): authenticate will time out at 5361.539371s 2024-03-25T17:30:08.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.293+0000 7f57056bf640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f57000a98f0 0x7f57000a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:33090/0 (socket says 172.21.15.134:33090) 2024-03-25T17:30:08.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.293+0000 7f57056bf640 1 -- 172.21.15.134:0/1384607903 learned_addr learned my addr 172.21.15.134:0/1384607903 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:08.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.305+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359744573 0 0) 0x7f57000b2a30 con 0x7f57001430c0 2024-03-25T17:30:08.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.305+0000 7f56f7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:08.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.305+0000 7f56f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:08.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.305+0000 7f56f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:08.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.305+0000 7f56f7fff640 10 monclient(hunting): my global_id is 34232 2024-03-25T17:30:08.306 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.305+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ec003520 con 0x7f57001430c0 2024-03-25T17:30:08.309 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.309+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3587492348 0 0) 0x7f57000b28a0 con 0x7f57000a98f0 2024-03-25T17:30:08.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.309+0000 7f56f7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:08.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.309+0000 7f56f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:08.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.309+0000 7f56f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:08.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.309+0000 7f56f7fff640 10 monclient(hunting): my global_id is 4252 2024-03-25T17:30:08.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.309+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57000b2a30 con 0x7f57000a98f0 2024-03-25T17:30:08.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1631480309 0 0) 0x7f57000b2a30 con 0x7f57000a98f0 2024-03-25T17:30:08.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 >> v1:172.21.15.134:6790/0 conn(0x7f57001430c0 legacy=0x7f570014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:08.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 >> v1:172.21.15.134:6789/0 conn(0x7f570013f710 legacy=0x7f57001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:08.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:30:08.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 1 monclient: found mon.noname-c 2024-03-25T17:30:08.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57000b44c0 con 0x7f57000a98f0 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 10 monclient: _finish_auth 0 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f56fc001670 con 0x7f57000a98f0 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f56f7fff640 10 monclient: dump: 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.318 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:08.319 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:08.319 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:08.319 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:08.319 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:08.319 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:08.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f5707923640 5 monclient: authenticate success, global_id 4252 2024-03-25T17:30:08.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.316+0000 7f5707923640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:08.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56fc0017d0 con 0x7f57000a98f0 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f56f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f56f7fff640 1 -- 172.21.15.134:0/1384607903 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f56fc001c20 con 0x7f57000a98f0 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f5707923640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f5707923640 4 set_mon_vals no callback set 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f56f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f56f7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.344+0000 7f56f7fff640 10 monclient: dump: 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:08.345 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:08.346 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:08.346 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:08.346 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:08.346 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.345+0000 7f5707923640 10 monclient: shutdown 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.345+0000 7f5707923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.345+0000 7f5707923640 1 -- 172.21.15.134:0/1384607903 >> v1:172.21.15.159:6789/0 conn(0x7f57000a98f0 legacy=0x7f57000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.345+0000 7f5707923640 1 -- 172.21.15.134:0/1384607903 shutdown_connections 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.345+0000 7f5707923640 1 -- 172.21.15.134:0/1384607903 >> 172.21.15.134:0/1384607903 conn(0x7f57000b7ea0 msgr2=0x7f57000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.345+0000 7f5707923640 1 -- 172.21.15.134:0/1384607903 shutdown_connections 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.345+0000 7f5707923640 1 -- 172.21.15.134:0/1384607903 wait complete. 2024-03-25T17:30:08.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 10 monclient: build_initial_monmap 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 10 monclient: monmap: 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 1 librados: starting msgr at 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 1 librados: starting objecter 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 1 Processor -- start 2024-03-25T17:30:08.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 1 -- start start 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 1 librados: setting wanted keys 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 1 librados: calling monclient init 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.346+0000 7f5707923640 10 monclient: init 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 10 monclient(hunting): picked mon.noname-b con 0x7f57000a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 10 monclient(hunting): picked mon.noname-c con 0x7f570013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 10 monclient(hunting): picked mon.noname-a con 0x7f57001430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57001390e0 con 0x7f57001430c0 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57001349e0 con 0x7f57000a98f0 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5700139350 con 0x7f570013f710 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:08.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.347+0000 7f5707923640 10 monclient(hunting): authenticate will time out at 5361.610010s 2024-03-25T17:30:08.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.356+0000 7f5704ebe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f570013f710 0x7f5700133bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:58808/0 (socket says 172.21.15.134:58808) 2024-03-25T17:30:08.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.356+0000 7f5704ebe640 1 -- 172.21.15.134:0/175298676 learned_addr learned my addr 172.21.15.134:0/175298676 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:08.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.357+0000 7f56f5ffb640 10 client.?.objecter ms_handle_connect 0x7f570013f710 2024-03-25T17:30:08.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.357+0000 7f56f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:08.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.365+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844295437 0 0) 0x7f5700139350 con 0x7f570013f710 2024-03-25T17:30:08.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.365+0000 7f56f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:08.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.365+0000 7f56f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:08.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.365+0000 7f56f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:08.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.365+0000 7f56f5ffb640 10 monclient(hunting): my global_id is 34235 2024-03-25T17:30:08.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.365+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e4004eb0 con 0x7f570013f710 2024-03-25T17:30:08.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.370+0000 7f56f5ffb640 10 client.?.objecter ms_handle_connect 0x7f57000a98f0 2024-03-25T17:30:08.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.370+0000 7f56f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:08.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.375+0000 7f56f5ffb640 10 client.?.objecter ms_handle_connect 0x7f57001430c0 2024-03-25T17:30:08.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.375+0000 7f56f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:08.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.385+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684852080 0 0) 0x7f57001349e0 con 0x7f57000a98f0 2024-03-25T17:30:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.385+0000 7f56f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.385+0000 7f56f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.385+0000 7f56f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.385+0000 7f56f5ffb640 10 monclient(hunting): my global_id is 4255 2024-03-25T17:30:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.385+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5700139350 con 0x7f57000a98f0 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.397+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4206250801 0 0) 0x7f56e4004eb0 con 0x7f570013f710 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.397+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57001349e0 con 0x7f570013f710 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.397+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f56f0001a30 con 0x7f570013f710 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.397+0000 7f56f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.397+0000 7f56f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.397+0000 7f56f5ffb640 10 monclient(hunting): dump: 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.398 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.397+0000 7f56f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.399+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2328159745 0 0) 0x7f57001390e0 con 0x7f57001430c0 2024-03-25T17:30:08.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.399+0000 7f56f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.399+0000 7f56f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.399+0000 7f56f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.399+0000 7f56f5ffb640 10 monclient(hunting): my global_id is 4245 2024-03-25T17:30:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.399+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e4004eb0 con 0x7f57001430c0 2024-03-25T17:30:08.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.407+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1458387652 0 0) 0x7f5700139350 con 0x7f57000a98f0 2024-03-25T17:30:08.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.407+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57001390e0 con 0x7f57000a98f0 2024-03-25T17:30:08.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.407+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f56fc001b20 con 0x7f57000a98f0 2024-03-25T17:30:08.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.407+0000 7f56f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.407+0000 7f56f5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.407+0000 7f56f5ffb640 10 monclient(hunting): dump: 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:08.408 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:08.409 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:08.409 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:08.409 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:08.409 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3034302247 0 0) 0x7f57001349e0 con 0x7f570013f710 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 >> v1:172.21.15.159:6789/0 conn(0x7f57000a98f0 legacy=0x7f57001d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 >> v1:172.21.15.134:6789/0 conn(0x7f57001430c0 legacy=0x7f57001342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 1 monclient: found mon.c 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5700139630 con 0x7f570013f710 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.414+0000 7f56f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:08.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 5 monclient: authenticate success, global_id 34235 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 10 monclient: _renew_subs 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5700139910 con 0x7f570013f710 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 10 client.34235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 10 monclient: _renew_subs 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5700137800 con 0x7f570013f710 2024-03-25T17:30:08.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.415+0000 7f5707923640 1 librados: init done 2024-03-25T17:30:08.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.416+0000 7f5707923640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:08.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.416+0000 7f5707923640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:08.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.416+0000 7f5707923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:08.418 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.416+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5700138410 con 0x7f570013f710 2024-03-25T17:30:08.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.722+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56f0001c70 con 0x7f570013f710 2024-03-25T17:30:08.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.722+0000 7f56f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:08.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.722+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f56f0003a60 con 0x7f570013f710 2024-03-25T17:30:08.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.722+0000 7f56f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:08.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.722+0000 7f56f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.723+0000 7f56f5ffb640 10 monclient: dump: 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:08.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.722+0000 7f56f77fe640 4 set_mon_vals no callback set 2024-03-25T17:30:08.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.723+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f56f0003c80 con 0x7f570013f710 2024-03-25T17:30:08.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.724+0000 7f57056bf640 10 monclient: get_auth_request con 0x7f56e406dfb0 auth_method 0 2024-03-25T17:30:08.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.724+0000 7f56f5ffb640 10 client.34235.objecter ms_handle_connect 0x7f56e406dfb0 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f56f0054880 con 0x7f570013f710 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 10 client.34235.objecter ms_dispatch 0x7f5700000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 3 client.34235.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 3 client.34235.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 20 client.34235.objecter dump_active .. 0 homeless 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56f0050390 con 0x7f570013f710 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:08.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.730+0000 7f56f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:08.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.951+0000 7f5707923640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:08.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.951+0000 7f5707923640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:08.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.951+0000 7f5707923640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:08.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.951+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5700074cc0 con 0x7f570013f710 2024-03-25T17:30:08.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.981+0000 7f56f5ffb640 1 -- 172.21.15.134:0/175298676 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f56f00522c0 con 0x7f570013f710 2024-03-25T17:30:08.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.981+0000 7f56f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:08.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.981+0000 7f56f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:08.982 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:08.983 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:08.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 10 librados: watch_flush enter 2024-03-25T17:30:08.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 10 librados: watch_flush exit 2024-03-25T17:30:08.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 20 client.34235.objecter shutdown clearing up homeless session... 2024-03-25T17:30:08.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 10 client.34235.objecter successfully canceled tick 2024-03-25T17:30:08.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 >> v1:172.21.15.134:6800/610837766 conn(0x7f56e406dfb0 legacy=0x7f56e4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:08.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 10 monclient: shutdown 2024-03-25T17:30:08.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:08.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 >> v1:172.21.15.134:6790/0 conn(0x7f570013f710 legacy=0x7f5700133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:08.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 shutdown_connections 2024-03-25T17:30:08.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 >> 172.21.15.134:0/175298676 conn(0x7f57000b7ea0 msgr2=0x7f5700074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:08.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 shutdown_connections 2024-03-25T17:30:08.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 1 -- 172.21.15.134:0/175298676 wait complete. 2024-03-25T17:30:08.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:08.984+0000 7f5707923640 1 librados: shutdown 2024-03-25T17:30:08.997 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:15.802 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:15.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.914+0000 7fe934406640 10 monclient: get_monmap_and_config 2024-03-25T17:30:15.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.914+0000 7fe934406640 10 monclient: build_initial_monmap 2024-03-25T17:30:15.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.914+0000 7fe934406640 10 monclient: monmap: 2024-03-25T17:30:15.914 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:15.914 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:15.915020+0000 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:15.915020+0000 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:15.915 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.914+0000 7fe934406640 1 Processor -- start 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.914+0000 7fe934406640 1 -- start start 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.914+0000 7fe934406640 10 monclient: init 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 10 monclient(hunting): picked mon.noname-c con 0x7fe92c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 10 monclient(hunting): picked mon.noname-b con 0x7fe92c0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 10 monclient(hunting): picked mon.noname-a con 0x7fe92c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c0af940 con 0x7fe92c14be90 2024-03-25T17:30:15.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c0b0a70 con 0x7fe92c0b4d60 2024-03-25T17:30:15.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c0b0d50 con 0x7fe92c0b1730 2024-03-25T17:30:15.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:15.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.915+0000 7fe934406640 10 monclient(hunting): authenticate will time out at 5369.178561s 2024-03-25T17:30:15.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.922+0000 7fe9329a3640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fe92c14be90 0x7fe92c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:59156/0 (socket says 172.21.15.134:59156) 2024-03-25T17:30:15.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.922+0000 7fe9329a3640 1 -- 172.21.15.134:0/2988434415 learned_addr learned my addr 172.21.15.134:0/2988434415 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:15.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.948+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659173821 0 0) 0x7fe92c0b0a70 con 0x7fe92c0b4d60 2024-03-25T17:30:15.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.948+0000 7fe9311a0640 10 monclient(hunting): _negotiate 2024-03-25T17:30:15.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.948+0000 7fe9311a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:15.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.948+0000 7fe9311a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:15.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.948+0000 7fe9311a0640 10 monclient(hunting): my global_id is 4258 2024-03-25T17:30:15.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.948+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe918003520 con 0x7fe92c0b4d60 2024-03-25T17:30:15.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.949+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197056740 0 0) 0x7fe92c0af940 con 0x7fe92c14be90 2024-03-25T17:30:15.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.949+0000 7fe9311a0640 10 monclient(hunting): _negotiate 2024-03-25T17:30:15.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.949+0000 7fe9311a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:15.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.949+0000 7fe9311a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:15.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.949+0000 7fe9311a0640 10 monclient(hunting): my global_id is 4248 2024-03-25T17:30:15.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.949+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c0b0a70 con 0x7fe92c14be90 2024-03-25T17:30:15.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.951+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970243945 0 0) 0x7fe92c0b0d50 con 0x7fe92c0b1730 2024-03-25T17:30:15.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.951+0000 7fe9311a0640 10 monclient(hunting): _negotiate 2024-03-25T17:30:15.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.951+0000 7fe9311a0640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:15.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.951+0000 7fe9311a0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:15.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.951+0000 7fe9311a0640 10 monclient(hunting): my global_id is 34238 2024-03-25T17:30:15.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.951+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c0af940 con 0x7fe92c0b1730 2024-03-25T17:30:15.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2506746318 0 0) 0x7fe918003520 con 0x7fe92c0b4d60 2024-03-25T17:30:15.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 >> v1:172.21.15.134:6790/0 conn(0x7fe92c0b1730 legacy=0x7fe92c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 >> v1:172.21.15.134:6789/0 conn(0x7fe92c14be90 legacy=0x7fe92c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 10 monclient: _finish_hunting 0 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 1 monclient: found mon.noname-c 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c0b1030 con 0x7fe92c0b4d60 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 10 monclient: _finish_auth 0 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 10 monclient: _check_auth_tickets 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe928001690 con 0x7fe92c0b4d60 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe9311a0640 10 monclient: dump: 2024-03-25T17:30:15.967 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe934406640 5 monclient: authenticate success, global_id 4258 2024-03-25T17:30:15.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.966+0000 7fe934406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:15.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.988+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9280017f0 con 0x7fe92c0b4d60 2024-03-25T17:30:15.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.988+0000 7fe9311a0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:15.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe9311a0640 1 -- 172.21.15.134:0/2988434415 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe928001c00 con 0x7fe92c0b4d60 2024-03-25T17:30:15.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:15.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 4 set_mon_vals no callback set 2024-03-25T17:30:15.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe9311a0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:15.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe9311a0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe9311a0640 10 monclient: dump: 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:15.990 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:15.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 10 monclient: shutdown 2024-03-25T17:30:15.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:15.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 1 -- 172.21.15.134:0/2988434415 >> v1:172.21.15.159:6789/0 conn(0x7fe92c0b4d60 legacy=0x7fe92c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:15.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 1 -- 172.21.15.134:0/2988434415 shutdown_connections 2024-03-25T17:30:15.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 1 -- 172.21.15.134:0/2988434415 >> 172.21.15.134:0/2988434415 conn(0x7fe92c0617a0 msgr2=0x7fe92c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:15.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 1 -- 172.21.15.134:0/2988434415 shutdown_connections 2024-03-25T17:30:15.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.989+0000 7fe934406640 1 -- 172.21.15.134:0/2988434415 wait complete. 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 10 monclient: build_initial_monmap 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 10 monclient: monmap: 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 1 librados: starting msgr at 2024-03-25T17:30:15.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 1 librados: starting objecter 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 1 Processor -- start 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 1 -- start start 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 1 librados: setting wanted keys 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 1 librados: calling monclient init 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.990+0000 7fe934406640 10 monclient: init 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 10 monclient(hunting): picked mon.noname-b con 0x7fe92c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 10 monclient(hunting): picked mon.noname-c con 0x7fe92c0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 10 monclient(hunting): picked mon.noname-a con 0x7fe92c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c140d00 con 0x7fe92c14be90 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c141cc0 con 0x7fe92c0b1730 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c1d8470 con 0x7fe92c0b4d60 2024-03-25T17:30:15.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:15.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:15.991+0000 7fe934406640 10 monclient(hunting): authenticate will time out at 5369.254541s 2024-03-25T17:30:16.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.002+0000 7fe9319a1640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe92c0b4d60 0x7fe92c1d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:42426/0 (socket says 172.21.15.134:42426) 2024-03-25T17:30:16.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.002+0000 7fe9319a1640 1 -- 172.21.15.134:0/1488511315 learned_addr learned my addr 172.21.15.134:0/1488511315 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:16.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.007+0000 7fe922ffd640 10 client.?.objecter ms_handle_connect 0x7fe92c0b4d60 2024-03-25T17:30:16.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.007+0000 7fe922ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:16.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.008+0000 7fe922ffd640 10 client.?.objecter ms_handle_connect 0x7fe92c14be90 2024-03-25T17:30:16.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.008+0000 7fe922ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.011+0000 7fe922ffd640 10 client.?.objecter ms_handle_connect 0x7fe92c0b1730 2024-03-25T17:30:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.011+0000 7fe922ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.018+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449345792 0 0) 0x7fe92c140d00 con 0x7fe92c14be90 2024-03-25T17:30:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.019+0000 7fe922ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.019+0000 7fe922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.019+0000 7fe922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.019+0000 7fe922ffd640 10 monclient(hunting): my global_id is 4251 2024-03-25T17:30:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.019+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe910004eb0 con 0x7fe92c14be90 2024-03-25T17:30:16.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.025+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547246867 0 0) 0x7fe92c1d8470 con 0x7fe92c0b4d60 2024-03-25T17:30:16.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.025+0000 7fe922ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:16.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.025+0000 7fe922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:16.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.025+0000 7fe922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:16.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.025+0000 7fe922ffd640 10 monclient(hunting): my global_id is 34241 2024-03-25T17:30:16.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.025+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c140d00 con 0x7fe92c0b4d60 2024-03-25T17:30:16.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.031+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 541591170 0 0) 0x7fe92c141cc0 con 0x7fe92c0b1730 2024-03-25T17:30:16.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.031+0000 7fe922ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:30:16.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.031+0000 7fe922ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:16.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.031+0000 7fe922ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.031+0000 7fe922ffd640 10 monclient(hunting): my global_id is 4261 2024-03-25T17:30:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.031+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c1d8470 con 0x7fe92c0b1730 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.034+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1479002421 0 0) 0x7fe910004eb0 con 0x7fe92c14be90 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.034+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe92c141cc0 con 0x7fe92c14be90 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.034+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe924001bd0 con 0x7fe92c14be90 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.034+0000 7fe922ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.034+0000 7fe922ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.034+0000 7fe922ffd640 10 monclient(hunting): dump: 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:16.035 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:16.036 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:16.036 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:16.036 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:16.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.034+0000 7fe922ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 365898386 0 0) 0x7fe92c141cc0 con 0x7fe92c14be90 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 >> v1:172.21.15.134:6790/0 conn(0x7fe92c0b4d60 legacy=0x7fe92c1d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 >> v1:172.21.15.159:6789/0 conn(0x7fe92c0b1730 legacy=0x7fe92c1405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 1 monclient: found mon.a 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c1d8600 con 0x7fe92c14be90 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 10 monclient: _finish_auth 0 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:30:16.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 5 monclient: authenticate success, global_id 4251 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 10 monclient: _renew_subs 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c13b3c0 con 0x7fe92c14be90 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 10 monclient: _renew_subs 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c1425a0 con 0x7fe92c14be90 2024-03-25T17:30:16.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.048+0000 7fe934406640 1 librados: init done 2024-03-25T17:30:16.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.049+0000 7fe934406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:16.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.049+0000 7fe934406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:16.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.049+0000 7fe934406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:16.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.049+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c0b2270 con 0x7fe92c14be90 2024-03-25T17:30:16.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.346+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe924004b60 con 0x7fe92c14be90 2024-03-25T17:30:16.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.347+0000 7fe922ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:16.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.347+0000 7fe93099f640 4 set_mon_vals no callback set 2024-03-25T17:30:16.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.347+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe924002100 con 0x7fe92c14be90 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.347+0000 7fe922ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.347+0000 7fe922ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.347+0000 7fe922ffd640 10 monclient: dump: 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:16.348 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:16.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.348+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fe92401ad20 con 0x7fe92c14be90 2024-03-25T17:30:16.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.348+0000 7fe9321a2640 10 monclient: get_auth_request con 0x7fe91006ddf0 auth_method 0 2024-03-25T17:30:16.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.348+0000 7fe922ffd640 10 client.4251.objecter ms_handle_connect 0x7fe91006ddf0 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fe924054980 con 0x7fe92c14be90 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 10 client.4251.objecter ms_dispatch 0x7fe92c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9240502c0 con 0x7fe92c14be90 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:16.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.353+0000 7fe922ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:16.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.571+0000 7fe934406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:16.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.571+0000 7fe934406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:16.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.571+0000 7fe934406640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:30:16.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.571+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe92c0627b0 con 0x7fe92c14be90 2024-03-25T17:30:16.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.595+0000 7fe922ffd640 1 -- 172.21.15.134:0/1488511315 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fe9240523c0 con 0x7fe92c14be90 2024-03-25T17:30:16.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.595+0000 7fe922ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:16.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.595+0000 7fe922ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:16.596 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:16.597 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 10 librados: watch_flush enter 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 10 librados: watch_flush exit 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 10 client.4251.objecter successfully canceled tick 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 >> v1:172.21.15.134:6800/610837766 conn(0x7fe91006ddf0 legacy=0x7fe9100702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 10 monclient: shutdown 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 >> v1:172.21.15.134:6789/0 conn(0x7fe92c14be90 legacy=0x7fe92c134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 shutdown_connections 2024-03-25T17:30:16.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 >> 172.21.15.134:0/1488511315 conn(0x7fe92c0617a0 msgr2=0x7fe92c0b6f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:16.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 shutdown_connections 2024-03-25T17:30:16.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 1 -- 172.21.15.134:0/1488511315 wait complete. 2024-03-25T17:30:16.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:16.598+0000 7fe934406640 1 librados: shutdown 2024-03-25T17:30:16.612 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:19.101 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:24.117 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.210 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:24.226 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.237 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:24.265 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.272 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:24.288 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.301 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:24.330 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.354 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:24.354 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.447 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:24.463 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.533 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:24.562 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:19.567 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:24.584 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:23.418 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:23.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.523+0000 7febc55f4640 10 monclient: get_monmap_and_config 2024-03-25T17:30:23.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.523+0000 7febc55f4640 10 monclient: build_initial_monmap 2024-03-25T17:30:23.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.523+0000 7febc55f4640 10 monclient: monmap: 2024-03-25T17:30:23.524 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:23.524 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:23.524 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:23.524151+0000 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:23.524151+0000 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.523+0000 7febc55f4640 1 Processor -- start 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.523+0000 7febc55f4640 1 -- start start 2024-03-25T17:30:23.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.523+0000 7febc55f4640 10 monclient: init 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 10 monclient(hunting): picked mon.noname-a con 0x7febc00a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 10 monclient(hunting): picked mon.noname-c con 0x7febc013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 10 monclient(hunting): picked mon.noname-b con 0x7febc01430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc0142520 con 0x7febc00a98f0 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc00b28a0 con 0x7febc013f710 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc00b2a30 con 0x7febc01430c0 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.524+0000 7febc55f4640 10 monclient(hunting): authenticate will time out at 5376.787355s 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.525+0000 7febbf7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7febc01430c0 0x7febc014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:49622/0 (socket says 172.21.15.134:49622) 2024-03-25T17:30:23.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.525+0000 7febbf7fe640 1 -- 172.21.15.134:0/1743064118 learned_addr learned my addr 172.21.15.134:0/1743064118 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:23.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.549+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350933384 0 0) 0x7febc00b28a0 con 0x7febc013f710 2024-03-25T17:30:23.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.549+0000 7febbe7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:23.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.549+0000 7febbe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:23.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.549+0000 7febbe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:23.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.549+0000 7febbe7fc640 10 monclient(hunting): my global_id is 4264 2024-03-25T17:30:23.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.549+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac003520 con 0x7febc013f710 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2151093824 0 0) 0x7febac003520 con 0x7febc013f710 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 >> v1:172.21.15.134:6790/0 conn(0x7febc01430c0 legacy=0x7febc014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 >> v1:172.21.15.134:6789/0 conn(0x7febc00a98f0 legacy=0x7febc00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 1 monclient: found mon.noname-c 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc00b44c0 con 0x7febc013f710 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 10 monclient: _finish_auth 0 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:30:23.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7feba8001690 con 0x7febc013f710 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febbe7fc640 10 monclient: dump: 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:23.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febc55f4640 5 monclient: authenticate success, global_id 4264 2024-03-25T17:30:23.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.564+0000 7febc55f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.576+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feba80017f0 con 0x7febc013f710 2024-03-25T17:30:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.576+0000 7febbe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.576+0000 7febbe7fc640 1 -- 172.21.15.134:0/1743064118 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7feba8001c00 con 0x7febc013f710 2024-03-25T17:30:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febc55f4640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febc55f4640 4 set_mon_vals no callback set 2024-03-25T17:30:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febbe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:23.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febbe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febbe7fc640 10 monclient: dump: 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:23.578 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febc55f4640 10 monclient: shutdown 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febc55f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.577+0000 7febc55f4640 1 -- 172.21.15.134:0/1743064118 >> v1:172.21.15.159:6789/0 conn(0x7febc013f710 legacy=0x7febc01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.578+0000 7febc55f4640 1 -- 172.21.15.134:0/1743064118 shutdown_connections 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.578+0000 7febc55f4640 1 -- 172.21.15.134:0/1743064118 >> 172.21.15.134:0/1743064118 conn(0x7febc00b7ea0 msgr2=0x7febc00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.578+0000 7febc55f4640 1 -- 172.21.15.134:0/1743064118 shutdown_connections 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.578+0000 7febc55f4640 1 -- 172.21.15.134:0/1743064118 wait complete. 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 10 monclient: build_initial_monmap 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 10 monclient: monmap: 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:23.580 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 1 librados: starting msgr at 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 1 librados: starting objecter 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 1 Processor -- start 2024-03-25T17:30:23.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 1 -- start start 2024-03-25T17:30:23.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 1 librados: setting wanted keys 2024-03-25T17:30:23.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 1 librados: calling monclient init 2024-03-25T17:30:23.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.579+0000 7febc55f4640 10 monclient: init 2024-03-25T17:30:23.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:23.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:30:23.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 10 monclient(hunting): picked mon.noname-b con 0x7febc00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:23.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 10 monclient(hunting): picked mon.noname-a con 0x7febc013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:23.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 10 monclient(hunting): picked mon.noname-c con 0x7febc01430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:23.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc0137df0 con 0x7febc013f710 2024-03-25T17:30:23.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc01337a0 con 0x7febc00a98f0 2024-03-25T17:30:23.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc0133930 con 0x7febc01430c0 2024-03-25T17:30:23.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:23.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.580+0000 7febc55f4640 10 monclient(hunting): authenticate will time out at 5376.843251s 2024-03-25T17:30:23.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.596+0000 7febbeffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7febc00a98f0 0x7febc01392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:48702/0 (socket says 172.21.15.134:48702) 2024-03-25T17:30:23.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.596+0000 7febbeffd640 1 -- 172.21.15.134:0/2526155608 learned_addr learned my addr 172.21.15.134:0/2526155608 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:23.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.597+0000 7febb77fe640 10 client.?.objecter ms_handle_connect 0x7febc00a98f0 2024-03-25T17:30:23.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.597+0000 7febb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:23.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.599+0000 7febb77fe640 10 client.?.objecter ms_handle_connect 0x7febc013f710 2024-03-25T17:30:23.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.599+0000 7febb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:23.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.610+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921788185 0 0) 0x7febc0137df0 con 0x7febc013f710 2024-03-25T17:30:23.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.611+0000 7febb77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:30:23.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.611+0000 7febb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:23.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.611+0000 7febb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:23.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.611+0000 7febb77fe640 10 monclient(hunting): my global_id is 4254 2024-03-25T17:30:23.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.611+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feba0004eb0 con 0x7febc013f710 2024-03-25T17:30:23.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.621+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595471167 0 0) 0x7febc01337a0 con 0x7febc00a98f0 2024-03-25T17:30:23.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.621+0000 7febb77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:30:23.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.621+0000 7febb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:23.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.621+0000 7febb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:23.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.621+0000 7febb77fe640 10 monclient(hunting): my global_id is 4267 2024-03-25T17:30:23.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.621+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc0137df0 con 0x7febc00a98f0 2024-03-25T17:30:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.623+0000 7febb77fe640 10 client.?.objecter ms_handle_connect 0x7febc01430c0 2024-03-25T17:30:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.623+0000 7febb77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:23.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.632+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2782231217 0 0) 0x7febc0137df0 con 0x7febc00a98f0 2024-03-25T17:30:23.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.632+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc01337a0 con 0x7febc00a98f0 2024-03-25T17:30:23.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.632+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7febb00019f0 con 0x7febc00a98f0 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.632+0000 7febb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.632+0000 7febb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.632+0000 7febb77fe640 10 monclient(hunting): dump: 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:23.633 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:23.634 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:23.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.632+0000 7febb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:23.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.635+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736030787 0 0) 0x7febc0133930 con 0x7febc01430c0 2024-03-25T17:30:23.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.635+0000 7febb77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:30:23.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.635+0000 7febb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:23.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.635+0000 7febb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:23.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.635+0000 7febb77fe640 10 monclient(hunting): my global_id is 34247 2024-03-25T17:30:23.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.635+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc0137df0 con 0x7febc01430c0 2024-03-25T17:30:23.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.639+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724700451 0 0) 0x7feba0004eb0 con 0x7febc013f710 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.639+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc0133930 con 0x7febc013f710 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.639+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7feba8001bc0 con 0x7febc013f710 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.639+0000 7febb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.639+0000 7febb77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.639+0000 7febb77fe640 10 monclient(hunting): dump: 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:23.640 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:23.641 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:23.641 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:23.641 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:23.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 465589686 0 0) 0x7febc01337a0 con 0x7febc00a98f0 2024-03-25T17:30:23.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 >> v1:172.21.15.134:6790/0 conn(0x7febc01430c0 legacy=0x7febc0133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:23.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 >> v1:172.21.15.134:6789/0 conn(0x7febc013f710 legacy=0x7febc01376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 1 monclient: found mon.b 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc01367d0 con 0x7febc00a98f0 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 10 monclient: _finish_auth 0 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 5 monclient: authenticate success, global_id 4267 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 10 monclient: _renew_subs 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:23.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc01369e0 con 0x7febc00a98f0 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 10 monclient: _renew_subs 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febc013b640 con 0x7febc00a98f0 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.647+0000 7febc55f4640 1 librados: init done 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.648+0000 7febc55f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.648+0000 7febc55f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.648+0000 7febc55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:23.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.648+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc00627b0 con 0x7febc00a98f0 2024-03-25T17:30:23.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.915+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febb0003610 con 0x7febc00a98f0 2024-03-25T17:30:23.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.915+0000 7febb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:23.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.915+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7febb00039e0 con 0x7febc00a98f0 2024-03-25T17:30:23.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.915+0000 7febb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:23.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.915+0000 7febb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.915+0000 7febb77fe640 10 monclient: dump: 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:23.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.915+0000 7febbdffb640 4 set_mon_vals no callback set 2024-03-25T17:30:23.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.916+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7febb0003bc0 con 0x7febc00a98f0 2024-03-25T17:30:23.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.917+0000 7febb7fff640 10 monclient: get_auth_request con 0x7feba006df70 auth_method 0 2024-03-25T17:30:23.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.917+0000 7febb77fe640 10 client.4267.objecter ms_handle_connect 0x7feba006df70 2024-03-25T17:30:23.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.921+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7febb00548e0 con 0x7febc00a98f0 2024-03-25T17:30:23.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.921+0000 7febb77fe640 10 client.4267.objecter ms_dispatch 0x7febc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:23.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.921+0000 7febb77fe640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:23.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.921+0000 7febb77fe640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:23.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.921+0000 7febb77fe640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T17:30:23.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.922+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febb00503f0 con 0x7febc00a98f0 2024-03-25T17:30:23.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.922+0000 7febb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:23.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:23.922+0000 7febb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:24.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.167+0000 7febc55f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:24.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.167+0000 7febc55f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:24.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.167+0000 7febc55f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:24.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.167+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febc0074cc0 con 0x7febc00a98f0 2024-03-25T17:30:24.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.189+0000 7febb77fe640 1 -- 172.21.15.134:0/2526155608 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7febb0052320 con 0x7febc00a98f0 2024-03-25T17:30:24.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.189+0000 7febb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:24.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.189+0000 7febb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:24.190 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:24.190 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 10 librados: watch_flush enter 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 10 librados: watch_flush exit 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 10 client.4267.objecter successfully canceled tick 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 >> v1:172.21.15.134:6800/610837766 conn(0x7feba006df70 legacy=0x7feba0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 10 monclient: shutdown 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:24.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 >> v1:172.21.15.159:6789/0 conn(0x7febc00a98f0 legacy=0x7febc01392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:24.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 shutdown_connections 2024-03-25T17:30:24.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.191+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 >> 172.21.15.134:0/2526155608 conn(0x7febc00b7ea0 msgr2=0x7febc0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:24.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.192+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 shutdown_connections 2024-03-25T17:30:24.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.192+0000 7febc55f4640 1 -- 172.21.15.134:0/2526155608 wait complete. 2024-03-25T17:30:24.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:24.192+0000 7febc55f4640 1 librados: shutdown 2024-03-25T17:30:24.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:31.012 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:31.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.106+0000 7efd1b425640 10 monclient: get_monmap_and_config 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.106+0000 7efd1b425640 10 monclient: build_initial_monmap 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.106+0000 7efd1b425640 10 monclient: monmap: 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:31.107392+0000 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:31.107392+0000 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:31.107 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:31.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 1 Processor -- start 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 1 -- start start 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient: init 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient(hunting): picked mon.noname-a con 0x7efd140a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient(hunting): picked mon.noname-c con 0x7efd1413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient(hunting): picked mon.noname-b con 0x7efd141430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd14142520 con 0x7efd140a98f0 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140b28a0 con 0x7efd1413f710 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140b2a30 con 0x7efd141430c0 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:31.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.107+0000 7efd1b425640 10 monclient(hunting): authenticate will time out at 5384.370745s 2024-03-25T17:30:31.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.128+0000 7efd191c1640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7efd140a98f0 0x7efd140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:46668/0 (socket says 172.21.15.134:46668) 2024-03-25T17:30:31.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.128+0000 7efd191c1640 1 -- 172.21.15.134:0/1693395654 learned_addr learned my addr 172.21.15.134:0/1693395654 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:31.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.141+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498534487 0 0) 0x7efd14142520 con 0x7efd140a98f0 2024-03-25T17:30:31.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.141+0000 7efd0bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:31.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.141+0000 7efd0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:31.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.141+0000 7efd0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:31.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.141+0000 7efd0bfff640 10 monclient(hunting): my global_id is 4257 2024-03-25T17:30:31.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.141+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd00003520 con 0x7efd140a98f0 2024-03-25T17:30:31.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2678635777 0 0) 0x7efd00003520 con 0x7efd140a98f0 2024-03-25T17:30:31.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 >> v1:172.21.15.134:6790/0 conn(0x7efd141430c0 legacy=0x7efd1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 >> v1:172.21.15.159:6789/0 conn(0x7efd1413f710 legacy=0x7efd141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 1 monclient: found mon.noname-a 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd140b44c0 con 0x7efd140a98f0 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 10 monclient: _finish_auth 0 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7efd10001690 con 0x7efd140a98f0 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd0bfff640 10 monclient: dump: 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:31.153 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd1b425640 5 monclient: authenticate success, global_id 4257 2024-03-25T17:30:31.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.152+0000 7efd1b425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.154+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd10001860 con 0x7efd140a98f0 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.154+0000 7efd0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.154+0000 7efd0bfff640 1 -- 172.21.15.134:0/1693395654 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7efd10001c70 con 0x7efd140a98f0 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.154+0000 7efd0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.154+0000 7efd0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.154+0000 7efd0bfff640 10 monclient: dump: 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.155 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.155+0000 7efd1b425640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:31.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.155+0000 7efd1b425640 4 set_mon_vals no callback set 2024-03-25T17:30:31.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.155+0000 7efd1b425640 10 monclient: shutdown 2024-03-25T17:30:31.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.155+0000 7efd1b425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:31.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.155+0000 7efd1b425640 1 -- 172.21.15.134:0/1693395654 >> v1:172.21.15.134:6789/0 conn(0x7efd140a98f0 legacy=0x7efd140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:31.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.155+0000 7efd1b425640 1 -- 172.21.15.134:0/1693395654 shutdown_connections 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.155+0000 7efd1b425640 1 -- 172.21.15.134:0/1693395654 >> 172.21.15.134:0/1693395654 conn(0x7efd140b7ea0 msgr2=0x7efd140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.156+0000 7efd1b425640 1 -- 172.21.15.134:0/1693395654 shutdown_connections 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.156+0000 7efd1b425640 1 -- 172.21.15.134:0/1693395654 wait complete. 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.156+0000 7efd1b425640 10 monclient: build_initial_monmap 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.156+0000 7efd1b425640 10 monclient: monmap: 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:31.158 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:31.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 librados: starting msgr at 2024-03-25T17:30:31.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 librados: starting objecter 2024-03-25T17:30:31.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 Processor -- start 2024-03-25T17:30:31.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 -- start start 2024-03-25T17:30:31.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 librados: setting wanted keys 2024-03-25T17:30:31.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 librados: calling monclient init 2024-03-25T17:30:31.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient: init 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient(hunting): picked mon.noname-b con 0x7efd140a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient(hunting): picked mon.noname-c con 0x7efd1413f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient(hunting): picked mon.noname-a con 0x7efd141430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd140ab8b0 con 0x7efd141430c0 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd14138740 con 0x7efd140a98f0 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd14134690 con 0x7efd1413f710 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:31.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.157+0000 7efd1b425640 10 monclient(hunting): authenticate will time out at 5384.420649s 2024-03-25T17:30:31.176 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.176+0000 7efd189c0640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7efd1413f710 0x7efd14133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:56462/0 (socket says 172.21.15.134:56462) 2024-03-25T17:30:31.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.176+0000 7efd189c0640 1 -- 172.21.15.134:0/977323581 learned_addr learned my addr 172.21.15.134:0/977323581 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:31.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.177+0000 7efd09ffb640 10 client.?.objecter ms_handle_connect 0x7efd1413f710 2024-03-25T17:30:31.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.177+0000 7efd09ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:31.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.183+0000 7efd09ffb640 10 client.?.objecter ms_handle_connect 0x7efd140a98f0 2024-03-25T17:30:31.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.183+0000 7efd09ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:31.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.183+0000 7efd09ffb640 10 client.?.objecter ms_handle_connect 0x7efd141430c0 2024-03-25T17:30:31.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.183+0000 7efd09ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:31.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.184+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074331615 0 0) 0x7efd14134690 con 0x7efd1413f710 2024-03-25T17:30:31.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.184+0000 7efd09ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:31.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.184+0000 7efd09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:31.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.184+0000 7efd09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:31.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.184+0000 7efd09ffb640 10 monclient(hunting): my global_id is 34250 2024-03-25T17:30:31.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.184+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcf8004eb0 con 0x7efd1413f710 2024-03-25T17:30:31.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.189+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744454336 0 0) 0x7efd14138740 con 0x7efd140a98f0 2024-03-25T17:30:31.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.189+0000 7efd09ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:31.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.189+0000 7efd09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:31.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.189+0000 7efd09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:31.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.189+0000 7efd09ffb640 10 monclient(hunting): my global_id is 4273 2024-03-25T17:30:31.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.189+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd14134690 con 0x7efd140a98f0 2024-03-25T17:30:31.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.191+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203588040 0 0) 0x7efd140ab8b0 con 0x7efd141430c0 2024-03-25T17:30:31.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.191+0000 7efd09ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:30:31.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.191+0000 7efd09ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:31.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.191+0000 7efd09ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:31.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.191+0000 7efd09ffb640 10 monclient(hunting): my global_id is 4260 2024-03-25T17:30:31.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.191+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd14138740 con 0x7efd141430c0 2024-03-25T17:30:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.199+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3580467797 0 0) 0x7efcf8004eb0 con 0x7efd1413f710 2024-03-25T17:30:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.199+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd140ab8b0 con 0x7efd1413f710 2024-03-25T17:30:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.199+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7efd04001a30 con 0x7efd1413f710 2024-03-25T17:30:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.199+0000 7efd09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.199+0000 7efd09ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:30:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.199+0000 7efd09ffb640 10 monclient(hunting): dump: 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:31.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.199+0000 7efd09ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:31.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.211+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2229331433 0 0) 0x7efd14134690 con 0x7efd140a98f0 2024-03-25T17:30:31.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.211+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcf8004eb0 con 0x7efd140a98f0 2024-03-25T17:30:31.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.211+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7efd10001b70 con 0x7efd140a98f0 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.211+0000 7efd09ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.211+0000 7efd09ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.211+0000 7efd09ffb640 10 monclient(hunting): dump: 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:31.212 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 500287365 0 0) 0x7efd140ab8b0 con 0x7efd1413f710 2024-03-25T17:30:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 >> v1:172.21.15.159:6789/0 conn(0x7efd140a98f0 legacy=0x7efd141377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 >> v1:172.21.15.134:6789/0 conn(0x7efd141430c0 legacy=0x7efd141da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:30:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 1 monclient: found mon.c 2024-03-25T17:30:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd14134820 con 0x7efd1413f710 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 10 monclient: _finish_auth 0 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd09ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd1b425640 5 monclient: authenticate success, global_id 34250 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd1b425640 10 monclient: _renew_subs 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd1b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd14137fe0 con 0x7efd1413f710 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd1b425640 10 client.34250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.215+0000 7efd1b425640 10 monclient: _renew_subs 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.216+0000 7efd1b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.216+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd141384c0 con 0x7efd1413f710 2024-03-25T17:30:31.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.216+0000 7efd1b425640 1 librados: init done 2024-03-25T17:30:31.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.217+0000 7efd1b425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:31.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.217+0000 7efd1b425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:31.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.217+0000 7efd1b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:31.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.217+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd140b3ef0 con 0x7efd1413f710 2024-03-25T17:30:31.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.504+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efd04001c70 con 0x7efd1413f710 2024-03-25T17:30:31.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.504+0000 7efd09ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.504+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7efd04003a60 con 0x7efd1413f710 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.504+0000 7efd09ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.504+0000 7efd09ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.504+0000 7efd09ffb640 10 monclient: dump: 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:31.505 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:31.506 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:31.506 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:31.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.504+0000 7efd0b7fe640 4 set_mon_vals no callback set 2024-03-25T17:30:31.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.505+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7efd04003c40 con 0x7efd1413f710 2024-03-25T17:30:31.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.505+0000 7efd191c1640 10 monclient: get_auth_request con 0x7efcf806df70 auth_method 0 2024-03-25T17:30:31.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.505+0000 7efd09ffb640 10 client.34250.objecter ms_handle_connect 0x7efcf806df70 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7efd040548b0 con 0x7efd1413f710 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 10 client.34250.objecter ms_dispatch 0x7efd14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 3 client.34250.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 3 client.34250.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 20 client.34250.objecter dump_active .. 0 homeless 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efd040503c0 con 0x7efd1413f710 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:31.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.512+0000 7efd09ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:31.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.757+0000 7efd1b425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:31.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.757+0000 7efd1b425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:31.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.757+0000 7efd1b425640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:31.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.757+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd141388d0 con 0x7efd1413f710 2024-03-25T17:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.782+0000 7efd09ffb640 1 -- 172.21.15.134:0/977323581 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7efd040522f0 con 0x7efd1413f710 2024-03-25T17:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.782+0000 7efd09ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:31.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.782+0000 7efd09ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:31.783 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:31.783 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:31.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 10 librados: watch_flush enter 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 10 librados: watch_flush exit 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 20 client.34250.objecter shutdown clearing up homeless session... 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 10 client.34250.objecter successfully canceled tick 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 >> v1:172.21.15.134:6800/610837766 conn(0x7efcf806df70 legacy=0x7efcf8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 10 monclient: shutdown 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.784+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 >> v1:172.21.15.134:6790/0 conn(0x7efd1413f710 legacy=0x7efd14133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.785+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 shutdown_connections 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.785+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 >> 172.21.15.134:0/977323581 conn(0x7efd140b7ea0 msgr2=0x7efd1413d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.785+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 shutdown_connections 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.785+0000 7efd1b425640 1 -- 172.21.15.134:0/977323581 wait complete. 2024-03-25T17:30:31.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:31.785+0000 7efd1b425640 1 librados: shutdown 2024-03-25T17:30:31.799 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:38.605 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:38.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.717+0000 7f23d1f72640 10 monclient: get_monmap_and_config 2024-03-25T17:30:38.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.717+0000 7f23d1f72640 10 monclient: build_initial_monmap 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.717+0000 7f23d1f72640 10 monclient: monmap: 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:38.718220+0000 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:38.718220+0000 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:38.718 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.717+0000 7f23d1f72640 1 Processor -- start 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.717+0000 7f23d1f72640 1 -- start start 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.717+0000 7f23d1f72640 10 monclient: init 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 10 monclient(hunting): picked mon.noname-b con 0x7f23cc0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 10 monclient(hunting): picked mon.noname-a con 0x7f23cc0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 10 monclient(hunting): picked mon.noname-c con 0x7f23cc14be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0af940 con 0x7f23cc0b1730 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0b0a70 con 0x7f23cc14be90 2024-03-25T17:30:38.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc0b0d50 con 0x7f23cc0b4d60 2024-03-25T17:30:38.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:38.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.718+0000 7f23d1f72640 10 monclient(hunting): authenticate will time out at 5391.981327s 2024-03-25T17:30:38.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.719+0000 7f23cb7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f23cc0b4d60 0x7f23cc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:60982/0 (socket says 172.21.15.134:60982) 2024-03-25T17:30:38.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.719+0000 7f23cbfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f23cc14be90 0x7f23cc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:43698/0 (socket says 172.21.15.134:43698) 2024-03-25T17:30:38.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.719+0000 7f23cb7fe640 1 -- 172.21.15.134:0/422920096 learned_addr learned my addr 172.21.15.134:0/422920096 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:38.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.766+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426564539 0 0) 0x7f23cc0af940 con 0x7f23cc0b1730 2024-03-25T17:30:38.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.766+0000 7f23ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:38.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.766+0000 7f23ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:38.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.766+0000 7f23ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:38.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.766+0000 7f23ca7fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T17:30:38.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.766+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b4003520 con 0x7f23cc0b1730 2024-03-25T17:30:38.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.783+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487785040 0 0) 0x7f23cc0b0a70 con 0x7f23cc14be90 2024-03-25T17:30:38.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:38.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:38.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:38.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 10 monclient(hunting): my global_id is 4276 2024-03-25T17:30:38.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc0af940 con 0x7f23cc14be90 2024-03-25T17:30:38.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3315886412 0 0) 0x7f23b4003520 con 0x7f23cc0b1730 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 >> v1:172.21.15.134:6790/0 conn(0x7f23cc0b4d60 legacy=0x7f23cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 >> v1:172.21.15.159:6789/0 conn(0x7f23cc14be90 legacy=0x7f23cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 1 monclient: found mon.noname-a 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc0b1030 con 0x7f23cc0b1730 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.784+0000 7f23ca7fc640 10 monclient: _finish_auth 0 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23ca7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f23c0001690 con 0x7f23cc0b1730 2024-03-25T17:30:38.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23ca7fc640 10 monclient: dump: 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23d1f72640 5 monclient: authenticate success, global_id 4263 2024-03-25T17:30:38.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.785+0000 7f23d1f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:38.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c00017f0 con 0x7f23cc0b1730 2024-03-25T17:30:38.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23ca7fc640 1 -- 172.21.15.134:0/422920096 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f23c0001c00 con 0x7f23cc0b1730 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23d1f72640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23d1f72640 4 set_mon_vals no callback set 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.794+0000 7f23ca7fc640 10 monclient: dump: 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:38.795 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:38.796 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:38.796 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.795+0000 7f23d1f72640 10 monclient: shutdown 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.795+0000 7f23d1f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.795+0000 7f23d1f72640 1 -- 172.21.15.134:0/422920096 >> v1:172.21.15.134:6789/0 conn(0x7f23cc0b1730 legacy=0x7f23cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.795+0000 7f23d1f72640 1 -- 172.21.15.134:0/422920096 shutdown_connections 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.795+0000 7f23d1f72640 1 -- 172.21.15.134:0/422920096 >> 172.21.15.134:0/422920096 conn(0x7f23cc0617a0 msgr2=0x7f23cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.795+0000 7f23d1f72640 1 -- 172.21.15.134:0/422920096 shutdown_connections 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.795+0000 7f23d1f72640 1 -- 172.21.15.134:0/422920096 wait complete. 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 10 monclient: build_initial_monmap 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 10 monclient: monmap: 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:38.798 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 1 librados: starting msgr at 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 1 librados: starting objecter 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 1 Processor -- start 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 1 -- start start 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 1 librados: setting wanted keys 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 1 librados: calling monclient init 2024-03-25T17:30:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.796+0000 7f23d1f72640 10 monclient: init 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 10 monclient(hunting): picked mon.noname-c con 0x7f23cc0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 10 monclient(hunting): picked mon.noname-b con 0x7f23cc0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 10 monclient(hunting): picked mon.noname-a con 0x7f23cc14be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc145910 con 0x7f23cc14be90 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc135810 con 0x7f23cc0b4d60 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23cc1453f0 con 0x7f23cc0b1730 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.797+0000 7f23d1f72640 10 monclient(hunting): authenticate will time out at 5392.060160s 2024-03-25T17:30:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.810+0000 7f23cb7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f23cc0b1730 0x7f23cc1d7690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:32770/0 (socket says 172.21.15.134:32770) 2024-03-25T17:30:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.810+0000 7f23cb7fe640 1 -- 172.21.15.134:0/4009786743 learned_addr learned my addr 172.21.15.134:0/4009786743 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:38.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.813+0000 7f23b3fff640 10 client.?.objecter ms_handle_connect 0x7f23cc0b4d60 2024-03-25T17:30:38.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.813+0000 7f23b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:38.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.814+0000 7f23b3fff640 10 client.?.objecter ms_handle_connect 0x7f23cc0b1730 2024-03-25T17:30:38.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.814+0000 7f23b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:38.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.824+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437807566 0 0) 0x7f23cc1453f0 con 0x7f23cc0b1730 2024-03-25T17:30:38.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.824+0000 7f23b3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:38.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.824+0000 7f23b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:38.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.824+0000 7f23b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:38.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.824+0000 7f23b3fff640 10 monclient(hunting): my global_id is 34256 2024-03-25T17:30:38.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.824+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b8004eb0 con 0x7f23cc0b1730 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.825+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515553445 0 0) 0x7f23cc135810 con 0x7f23cc0b4d60 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.825+0000 7f23b3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.825+0000 7f23b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.825+0000 7f23b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.825+0000 7f23b3fff640 10 monclient(hunting): my global_id is 4279 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.825+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc1453f0 con 0x7f23cc0b4d60 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.826+0000 7f23b3fff640 10 client.?.objecter ms_handle_connect 0x7f23cc14be90 2024-03-25T17:30:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.826+0000 7f23b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:38.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.837+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847981454 0 0) 0x7f23cc145910 con 0x7f23cc14be90 2024-03-25T17:30:38.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.837+0000 7f23b3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:30:38.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.837+0000 7f23b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:38.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.837+0000 7f23b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:38.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.837+0000 7f23b3fff640 10 monclient(hunting): my global_id is 4266 2024-03-25T17:30:38.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.837+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23cc135810 con 0x7f23cc14be90 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.840+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4031639798 0 0) 0x7f23cc1453f0 con 0x7f23cc0b4d60 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.840+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23cc145910 con 0x7f23cc0b4d60 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.840+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f23c0001bc0 con 0x7f23cc0b4d60 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.840+0000 7f23b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.840+0000 7f23b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.840+0000 7f23b3fff640 10 monclient(hunting): dump: 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:38.841 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:38.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.840+0000 7f23b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.847+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3574023955 0 0) 0x7f23b8004eb0 con 0x7f23cc0b1730 2024-03-25T17:30:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.847+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23cc1453f0 con 0x7f23cc0b1730 2024-03-25T17:30:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.847+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f23bc0018b0 con 0x7f23cc0b1730 2024-03-25T17:30:38.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.847+0000 7f23b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.847+0000 7f23b3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.847+0000 7f23b3fff640 10 monclient(hunting): dump: 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:38.848 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:38.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.853+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1350332264 0 0) 0x7f23cc1453f0 con 0x7f23cc0b1730 2024-03-25T17:30:38.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 >> v1:172.21.15.159:6789/0 conn(0x7f23cc0b4d60 legacy=0x7f23cc1349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 >> v1:172.21.15.134:6789/0 conn(0x7f23cc14be90 legacy=0x7f23cc135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 1 monclient: found mon.c 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23cc137bc0 con 0x7f23cc0b1730 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 10 monclient: _finish_auth 0 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 5 monclient: authenticate success, global_id 34256 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 10 monclient: _renew_subs 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:38.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23cc1405f0 con 0x7f23cc0b1730 2024-03-25T17:30:38.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 10 client.34256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:38.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 10 monclient: _renew_subs 2024-03-25T17:30:38.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:38.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23cc1d7f80 con 0x7f23cc0b1730 2024-03-25T17:30:38.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.854+0000 7f23d1f72640 1 librados: init done 2024-03-25T17:30:38.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.855+0000 7f23d1f72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:38.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.855+0000 7f23d1f72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:38.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.855+0000 7f23d1f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:38.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:38.855+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23cc140990 con 0x7f23cc0b1730 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23bc001a10 con 0x7f23cc0b1730 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f23bc0049c0 con 0x7f23cc0b1730 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23b3fff640 10 monclient: dump: 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:39.164 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23c9ffb640 4 set_mon_vals no callback set 2024-03-25T17:30:39.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.163+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f23bc004be0 con 0x7f23cc0b1730 2024-03-25T17:30:39.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.164+0000 7f23caffd640 10 monclient: get_auth_request con 0x7f23b806e000 auth_method 0 2024-03-25T17:30:39.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.164+0000 7f23b3fff640 10 client.34256.objecter ms_handle_connect 0x7f23b806e000 2024-03-25T17:30:39.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f23bc054870 con 0x7f23cc0b1730 2024-03-25T17:30:39.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 10 client.34256.objecter ms_dispatch 0x7f23cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:39.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 3 client.34256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:39.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 3 client.34256.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:39.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 20 client.34256.objecter dump_active .. 0 homeless 2024-03-25T17:30:39.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23bc0501b0 con 0x7f23cc0b1730 2024-03-25T17:30:39.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:39.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.172+0000 7f23b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:39.248 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:44.264 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.257 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:44.273 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.319 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:01:44.335 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.321 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:44.350 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.324 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:44.353 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.327 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:44.343 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.331 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:44.347 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.381+0000 7f23d1f72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:39.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.381+0000 7f23d1f72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:39.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.381+0000 7f23d1f72640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:30:39.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.381+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23cc145580 con 0x7f23cc0b1730 2024-03-25T17:30:39.389 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:01:44.418 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.392 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:01:44.408 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:30:39.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.458+0000 7f23b3fff640 1 -- 172.21.15.134:0/4009786743 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f23bc0522b0 con 0x7f23cc0b1730 2024-03-25T17:30:39.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.459+0000 7f23b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:39.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.459+0000 7f23b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:39.460 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:39.460 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 10 librados: watch_flush enter 2024-03-25T17:30:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 10 librados: watch_flush exit 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 20 client.34256.objecter shutdown clearing up homeless session... 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 10 client.34256.objecter successfully canceled tick 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 >> v1:172.21.15.134:6800/610837766 conn(0x7f23b806e000 legacy=0x7f23b80704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 10 monclient: shutdown 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 >> v1:172.21.15.134:6790/0 conn(0x7f23cc0b1730 legacy=0x7f23cc1d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 shutdown_connections 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.461+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 >> 172.21.15.134:0/4009786743 conn(0x7f23cc0617a0 msgr2=0x7f23cc13c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.462+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 shutdown_connections 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.462+0000 7f23d1f72640 1 -- 172.21.15.134:0/4009786743 wait complete. 2024-03-25T17:30:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:39.462+0000 7f23d1f72640 1 librados: shutdown 2024-03-25T17:30:39.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:46.279 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:46.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.398+0000 7fbba4027640 10 monclient: get_monmap_and_config 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.398+0000 7fbba4027640 10 monclient: build_initial_monmap 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.398+0000 7fbba4027640 10 monclient: monmap: 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:46.399041+0000 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:46.399041+0000 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.398+0000 7fbba4027640 1 Processor -- start 2024-03-25T17:30:46.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.398+0000 7fbba4027640 1 -- start start 2024-03-25T17:30:46.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.398+0000 7fbba4027640 10 monclient: init 2024-03-25T17:30:46.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.399+0000 7fbba4027640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:46.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.399+0000 7fbba4027640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.399+0000 7fbba4027640 10 monclient(hunting): picked mon.noname-b con 0x7fbb9c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.399+0000 7fbba4027640 10 monclient(hunting): picked mon.noname-a con 0x7fbb9c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.400+0000 7fbba4027640 10 monclient(hunting): picked mon.noname-c con 0x7fbb9c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.400+0000 7fbba4027640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c142520 con 0x7fbb9c13f710 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.400+0000 7fbba4027640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c0b28a0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.400+0000 7fbba4027640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c0b2a30 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.400+0000 7fbba4027640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:46.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.400+0000 7fbba4027640 10 monclient(hunting): authenticate will time out at 5399.662959s 2024-03-25T17:30:46.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.416+0000 7fbba15c2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbb9c13f710 0x7fbb9c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:53020/0 (socket says 172.21.15.134:53020) 2024-03-25T17:30:46.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.416+0000 7fbba15c2640 1 -- 172.21.15.134:0/2119461317 learned_addr learned my addr 172.21.15.134:0/2119461317 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:46.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.423+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735834029 0 0) 0x7fbb9c142520 con 0x7fbb9c13f710 2024-03-25T17:30:46.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.423+0000 7fbba0dc1640 10 monclient(hunting): _negotiate 2024-03-25T17:30:46.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.423+0000 7fbba0dc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:46.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.423+0000 7fbba0dc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:46.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.423+0000 7fbba0dc1640 10 monclient(hunting): my global_id is 4269 2024-03-25T17:30:46.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.423+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb88003520 con 0x7fbb9c13f710 2024-03-25T17:30:46.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.430+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937575036 0 0) 0x7fbb9c0b2a30 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.430+0000 7fbba0dc1640 10 monclient(hunting): _negotiate 2024-03-25T17:30:46.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.430+0000 7fbba0dc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.430+0000 7fbba0dc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.430+0000 7fbba0dc1640 10 monclient(hunting): my global_id is 34259 2024-03-25T17:30:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.430+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb9c142520 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.444+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155403251 0 0) 0x7fbb9c0b28a0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.444+0000 7fbba0dc1640 10 monclient(hunting): _negotiate 2024-03-25T17:30:46.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.444+0000 7fbba0dc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:46.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.444+0000 7fbba0dc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:46.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.444+0000 7fbba0dc1640 10 monclient(hunting): my global_id is 4282 2024-03-25T17:30:46.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.444+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb9c0b2a30 con 0x7fbb9c1430c0 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3817442462 0 0) 0x7fbb88003520 con 0x7fbb9c13f710 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 >> v1:172.21.15.134:6790/0 conn(0x7fbb9c0a98f0 legacy=0x7fbb9c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 >> v1:172.21.15.159:6789/0 conn(0x7fbb9c1430c0 legacy=0x7fbb9c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 10 monclient: _finish_hunting 0 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 1 monclient: found mon.noname-a 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:30:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c0b44c0 con 0x7fbb9c13f710 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 10 monclient: _finish_auth 0 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 10 monclient: _check_auth_tickets 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbb8c001690 con 0x7fbb9c13f710 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba0dc1640 10 monclient: dump: 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:46.454 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:46.455 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:46.455 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:46.455 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:46.455 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba4027640 5 monclient: authenticate success, global_id 4269 2024-03-25T17:30:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.452+0000 7fbba4027640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb8c0017f0 con 0x7fbb9c13f710 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba0dc1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba0dc1640 1 -- 172.21.15.134:0/2119461317 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbb8c001c00 con 0x7fbb9c13f710 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba4027640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba4027640 4 set_mon_vals no callback set 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba0dc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba0dc1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.462+0000 7fbba0dc1640 10 monclient: dump: 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:46.463 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:46.464 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.463+0000 7fbba4027640 10 monclient: shutdown 2024-03-25T17:30:46.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.463+0000 7fbba4027640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:46.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.463+0000 7fbba4027640 1 -- 172.21.15.134:0/2119461317 >> v1:172.21.15.134:6789/0 conn(0x7fbb9c13f710 legacy=0x7fbb9c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.463+0000 7fbba4027640 1 -- 172.21.15.134:0/2119461317 shutdown_connections 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.463+0000 7fbba4027640 1 -- 172.21.15.134:0/2119461317 >> 172.21.15.134:0/2119461317 conn(0x7fbb9c0b7ea0 msgr2=0x7fbb9c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.463+0000 7fbba4027640 1 -- 172.21.15.134:0/2119461317 shutdown_connections 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.464+0000 7fbba4027640 1 -- 172.21.15.134:0/2119461317 wait complete. 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.464+0000 7fbba4027640 10 monclient: build_initial_monmap 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.464+0000 7fbba4027640 10 monclient: monmap: 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:46.466 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.465+0000 7fbba4027640 1 librados: starting msgr at 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.465+0000 7fbba4027640 1 librados: starting objecter 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.465+0000 7fbba4027640 1 Processor -- start 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.465+0000 7fbba4027640 1 -- start start 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.465+0000 7fbba4027640 1 librados: setting wanted keys 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.465+0000 7fbba4027640 1 librados: calling monclient init 2024-03-25T17:30:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.465+0000 7fbba4027640 10 monclient: init 2024-03-25T17:30:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:30:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 10 monclient(hunting): picked mon.noname-c con 0x7fbb9c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 10 monclient(hunting): picked mon.noname-a con 0x7fbb9c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 10 monclient(hunting): picked mon.noname-b con 0x7fbb9c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c135550 con 0x7fbb9c13f710 2024-03-25T17:30:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c136b50 con 0x7fbb9c1430c0 2024-03-25T17:30:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c136e30 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.466+0000 7fbba4027640 10 monclient(hunting): authenticate will time out at 5399.728876s 2024-03-25T17:30:46.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.482+0000 7fbba25c4640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbb9c1430c0 0x7fbb9c1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:49298/0 (socket says 172.21.15.134:49298) 2024-03-25T17:30:46.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.482+0000 7fbba25c4640 1 -- 172.21.15.134:0/1670876544 learned_addr learned my addr 172.21.15.134:0/1670876544 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:46.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.483+0000 7fbb927fc640 10 client.?.objecter ms_handle_connect 0x7fbb9c1430c0 2024-03-25T17:30:46.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.483+0000 7fbb927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:46.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.488+0000 7fbb927fc640 10 client.?.objecter ms_handle_connect 0x7fbb9c13f710 2024-03-25T17:30:46.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.488+0000 7fbb927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:46.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.496+0000 7fbb927fc640 10 client.?.objecter ms_handle_connect 0x7fbb9c0a98f0 2024-03-25T17:30:46.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.496+0000 7fbb927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:46.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.497+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307395668 0 0) 0x7fbb9c136b50 con 0x7fbb9c1430c0 2024-03-25T17:30:46.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.497+0000 7fbb927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:46.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.497+0000 7fbb927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:46.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.497+0000 7fbb927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:46.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.497+0000 7fbb927fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T17:30:46.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.497+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb80004eb0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.501+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402287554 0 0) 0x7fbb9c135550 con 0x7fbb9c13f710 2024-03-25T17:30:46.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.501+0000 7fbb927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:46.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.501+0000 7fbb927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:46.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.501+0000 7fbb927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:46.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.501+0000 7fbb927fc640 10 monclient(hunting): my global_id is 4272 2024-03-25T17:30:46.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.501+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb9c136b50 con 0x7fbb9c13f710 2024-03-25T17:30:46.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.510+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849927501 0 0) 0x7fbb9c136e30 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.510+0000 7fbb927fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:46.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.510+0000 7fbb927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:46.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.510+0000 7fbb927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:46.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.510+0000 7fbb927fc640 10 monclient(hunting): my global_id is 34262 2024-03-25T17:30:46.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.510+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb9c135550 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.528+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1903071791 0 0) 0x7fbb80004eb0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.528+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb9c136e30 con 0x7fbb9c1430c0 2024-03-25T17:30:46.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.528+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbb940018f0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.528+0000 7fbb927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.528+0000 7fbb927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.528+0000 7fbb927fc640 10 monclient(hunting): dump: 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.528+0000 7fbb927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.536+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358405325 0 0) 0x7fbb9c136b50 con 0x7fbb9c13f710 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.536+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb80004eb0 con 0x7fbb9c13f710 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.536+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbb8c001b00 con 0x7fbb9c13f710 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.536+0000 7fbb927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.536+0000 7fbb927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.536+0000 7fbb927fc640 10 monclient(hunting): dump: 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:46.537 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:46.538 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:46.538 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:46.538 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.538+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1336037116 0 0) 0x7fbb9c135550 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.538+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb9c136b50 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.538+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbb98001cb0 con 0x7fbb9c0a98f0 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.538+0000 7fbb927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.538+0000 7fbb927fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.538+0000 7fbb927fc640 10 monclient(hunting): dump: 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:46.539 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:46.540 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1408127348 0 0) 0x7fbb9c136e30 con 0x7fbb9c1430c0 2024-03-25T17:30:46.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 >> v1:172.21.15.134:6790/0 conn(0x7fbb9c0a98f0 legacy=0x7fbb9c13b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:46.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 >> v1:172.21.15.134:6789/0 conn(0x7fbb9c13f710 legacy=0x7fbb9c1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:46.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 10 monclient: _finish_hunting 0 2024-03-25T17:30:46.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 1 monclient: found mon.b 2024-03-25T17:30:46.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:46.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c1357f0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 10 monclient: _finish_auth 0 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 10 monclient: _check_auth_tickets 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbb927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 5 monclient: authenticate success, global_id 4285 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 10 monclient: _renew_subs 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9c137cb0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 10 monclient: _renew_subs 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9c1352e0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.549+0000 7fbba4027640 1 librados: init done 2024-03-25T17:30:46.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.550+0000 7fbba4027640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:46.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.550+0000 7fbba4027640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:46.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.550+0000 7fbba4027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:46.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.551+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9c074cc0 con 0x7fbb9c1430c0 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.866+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb94003610 con 0x7fbb9c1430c0 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.866+0000 7fbb927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.866+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbb94003a20 con 0x7fbb9c1430c0 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.866+0000 7fbb927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.866+0000 7fbb927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.866+0000 7fbb927fc640 10 monclient: dump: 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:46.867 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:46.868 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:46.868 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:46.868 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:46.868 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:46.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.866+0000 7fbb93fff640 4 set_mon_vals no callback set 2024-03-25T17:30:46.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.867+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fbb94003c40 con 0x7fbb9c1430c0 2024-03-25T17:30:46.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.867+0000 7fbba1dc3640 10 monclient: get_auth_request con 0x7fbb8006e0f0 auth_method 0 2024-03-25T17:30:46.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.868+0000 7fbb927fc640 10 client.4285.objecter ms_handle_connect 0x7fbb8006e0f0 2024-03-25T17:30:46.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.869+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fbb94054740 con 0x7fbb9c1430c0 2024-03-25T17:30:46.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.869+0000 7fbb927fc640 10 client.4285.objecter ms_dispatch 0x7fbb9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:46.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.869+0000 7fbb927fc640 3 client.4285.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:46.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.869+0000 7fbb927fc640 3 client.4285.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:46.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.869+0000 7fbb927fc640 20 client.4285.objecter dump_active .. 0 homeless 2024-03-25T17:30:46.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.871+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb94050250 con 0x7fbb9c1430c0 2024-03-25T17:30:46.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.872+0000 7fbb927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:46.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:46.872+0000 7fbb927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:47.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.129+0000 7fbba4027640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:47.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.129+0000 7fbba4027640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:47.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.129+0000 7fbba4027640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:47.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.129+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb9c135980 con 0x7fbb9c1430c0 2024-03-25T17:30:47.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.296+0000 7fbb927fc640 1 -- 172.21.15.134:0/1670876544 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fbb94052180 con 0x7fbb9c1430c0 2024-03-25T17:30:47.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.296+0000 7fbb927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:47.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.296+0000 7fbb927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:47.297 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:47.297 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:47.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 10 librados: watch_flush enter 2024-03-25T17:30:47.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 10 librados: watch_flush exit 2024-03-25T17:30:47.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 20 client.4285.objecter shutdown clearing up homeless session... 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 10 client.4285.objecter successfully canceled tick 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 >> v1:172.21.15.134:6800/610837766 conn(0x7fbb8006e0f0 legacy=0x7fbb800705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 10 monclient: shutdown 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 >> v1:172.21.15.159:6789/0 conn(0x7fbb9c1430c0 legacy=0x7fbb9c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 shutdown_connections 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 >> 172.21.15.134:0/1670876544 conn(0x7fbb9c0b7ea0 msgr2=0x7fbb9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 shutdown_connections 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 1 -- 172.21.15.134:0/1670876544 wait complete. 2024-03-25T17:30:47.300 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:47.298+0000 7fbba4027640 1 librados: shutdown 2024-03-25T17:30:47.311 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:30:54.118 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:30:54.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.228+0000 7f0e4bc40640 10 monclient: get_monmap_and_config 2024-03-25T17:30:54.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.228+0000 7f0e4bc40640 10 monclient: build_initial_monmap 2024-03-25T17:30:54.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.228+0000 7f0e4bc40640 10 monclient: monmap: 2024-03-25T17:30:54.230 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:30:54.230935+0000 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:30:54.230935+0000 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:30:54.231 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:54.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.229+0000 7f0e4bc40640 1 Processor -- start 2024-03-25T17:30:54.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.230+0000 7f0e4bc40640 1 -- start start 2024-03-25T17:30:54.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.230+0000 7f0e4bc40640 10 monclient: init 2024-03-25T17:30:54.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.230+0000 7f0e4bc40640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:54.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.230+0000 7f0e4bc40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:30:54.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.230+0000 7f0e4bc40640 10 monclient(hunting): picked mon.noname-a con 0x7f0e440b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:54.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.230+0000 7f0e4bc40640 10 monclient(hunting): picked mon.noname-c con 0x7f0e440b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:54.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.230+0000 7f0e4bc40640 10 monclient(hunting): picked mon.noname-b con 0x7f0e4414be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:54.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.231+0000 7f0e4bc40640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e440af940 con 0x7f0e440b4d60 2024-03-25T17:30:54.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.231+0000 7f0e4bc40640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e440b0a70 con 0x7f0e440b1730 2024-03-25T17:30:54.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.231+0000 7f0e4bc40640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e440b0d50 con 0x7f0e4414be90 2024-03-25T17:30:54.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.231+0000 7f0e4bc40640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:54.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.231+0000 7f0e4bc40640 10 monclient(hunting): authenticate will time out at 5407.494800s 2024-03-25T17:30:54.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.242+0000 7f0e491db640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f0e440b1730 0x7f0e440b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:46260/0 (socket says 172.21.15.134:46260) 2024-03-25T17:30:54.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.242+0000 7f0e491db640 1 -- 172.21.15.134:0/2483372471 learned_addr learned my addr 172.21.15.134:0/2483372471 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:54.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.255+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443965438 0 0) 0x7f0e440b0d50 con 0x7f0e4414be90 2024-03-25T17:30:54.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.255+0000 7f0e489da640 10 monclient(hunting): _negotiate 2024-03-25T17:30:54.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.255+0000 7f0e489da640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:54.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.255+0000 7f0e489da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:54.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.255+0000 7f0e489da640 10 monclient(hunting): my global_id is 34265 2024-03-25T17:30:54.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.255+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e30003520 con 0x7f0e4414be90 2024-03-25T17:30:54.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.256+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653469707 0 0) 0x7f0e440af940 con 0x7f0e440b4d60 2024-03-25T17:30:54.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.256+0000 7f0e489da640 10 monclient(hunting): _negotiate 2024-03-25T17:30:54.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.256+0000 7f0e489da640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:54.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.256+0000 7f0e489da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:54.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.256+0000 7f0e489da640 10 monclient(hunting): my global_id is 4275 2024-03-25T17:30:54.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.256+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e440b0d50 con 0x7f0e440b4d60 2024-03-25T17:30:54.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.258+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472998180 0 0) 0x7f0e440b0a70 con 0x7f0e440b1730 2024-03-25T17:30:54.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.258+0000 7f0e489da640 10 monclient(hunting): _negotiate 2024-03-25T17:30:54.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.258+0000 7f0e489da640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:54.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.258+0000 7f0e489da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:54.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.258+0000 7f0e489da640 10 monclient(hunting): my global_id is 4288 2024-03-25T17:30:54.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.258+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e440af940 con 0x7f0e440b1730 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4097414402 0 0) 0x7f0e30003520 con 0x7f0e4414be90 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 >> v1:172.21.15.159:6789/0 conn(0x7f0e440b1730 legacy=0x7f0e440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 >> v1:172.21.15.134:6789/0 conn(0x7f0e440b4d60 legacy=0x7f0e4413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 10 monclient: _finish_hunting 0 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 1 monclient: found mon.noname-b 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e440b1030 con 0x7f0e4414be90 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 10 monclient: _finish_auth 0 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 10 monclient: _check_auth_tickets 2024-03-25T17:30:54.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0e3c001a10 con 0x7f0e4414be90 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e489da640 10 monclient: dump: 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:54.273 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e4bc40640 5 monclient: authenticate success, global_id 34265 2024-03-25T17:30:54.274 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.270+0000 7f0e4bc40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:30:54.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e3c001b70 con 0x7f0e4414be90 2024-03-25T17:30:54.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e489da640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e489da640 1 -- 172.21.15.134:0/2483372471 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0e3c001f80 con 0x7f0e4414be90 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e4bc40640 10 monclient: get_monmap_and_config success 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e4bc40640 4 set_mon_vals no callback set 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e489da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e489da640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.290+0000 7f0e489da640 10 monclient: dump: 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:54.292 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:54.293 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:54.293 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.291+0000 7f0e4bc40640 10 monclient: shutdown 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.291+0000 7f0e4bc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.291+0000 7f0e4bc40640 1 -- 172.21.15.134:0/2483372471 >> v1:172.21.15.134:6790/0 conn(0x7f0e4414be90 legacy=0x7f0e4414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.291+0000 7f0e4bc40640 1 -- 172.21.15.134:0/2483372471 shutdown_connections 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.291+0000 7f0e4bc40640 1 -- 172.21.15.134:0/2483372471 >> 172.21.15.134:0/2483372471 conn(0x7f0e440617a0 msgr2=0x7f0e44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.291+0000 7f0e4bc40640 1 -- 172.21.15.134:0/2483372471 shutdown_connections 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.291+0000 7f0e4bc40640 1 -- 172.21.15.134:0/2483372471 wait complete. 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 10 monclient: build_initial_monmap 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 10 monclient: monmap: 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:30:54.295 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 1 librados: starting msgr at 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 1 librados: starting objecter 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 1 Processor -- start 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 1 -- start start 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 1 librados: setting wanted keys 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 1 librados: calling monclient init 2024-03-25T17:30:54.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.292+0000 7f0e4bc40640 10 monclient: init 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 10 monclient: _reopen_session rank -1 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 10 monclient(hunting): picked mon.noname-c con 0x7f0e440b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 10 monclient(hunting): picked mon.noname-b con 0x7f0e440b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 10 monclient(hunting): picked mon.noname-a con 0x7f0e4414be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e44140ae0 con 0x7f0e4414be90 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e44135550 con 0x7f0e440b4d60 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e441356e0 con 0x7f0e440b1730 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 10 monclient(hunting): _renew_subs 2024-03-25T17:30:54.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.293+0000 7f0e4bc40640 10 monclient(hunting): authenticate will time out at 5407.557130s 2024-03-25T17:30:54.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.299+0000 7f0e491db640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f0e440b4d60 0x7f0e441db000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:46292/0 (socket says 172.21.15.134:46292) 2024-03-25T17:30:54.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.300+0000 7f0e491db640 1 -- 172.21.15.134:0/4251293306 learned_addr learned my addr 172.21.15.134:0/4251293306 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:30:54.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.302+0000 7f0e3a7fc640 10 client.?.objecter ms_handle_connect 0x7f0e440b4d60 2024-03-25T17:30:54.304 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.302+0000 7f0e3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:54.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.305+0000 7f0e3a7fc640 10 client.?.objecter ms_handle_connect 0x7f0e4414be90 2024-03-25T17:30:54.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.305+0000 7f0e3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:54.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.309+0000 7f0e3a7fc640 10 client.?.objecter ms_handle_connect 0x7f0e440b1730 2024-03-25T17:30:54.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.309+0000 7f0e3a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:30:54.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.309+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561360559 0 0) 0x7f0e44135550 con 0x7f0e440b4d60 2024-03-25T17:30:54.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.309+0000 7f0e3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:54.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.309+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:54.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.309+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:54.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.309+0000 7f0e3a7fc640 10 monclient(hunting): my global_id is 4291 2024-03-25T17:30:54.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.310+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e28004eb0 con 0x7f0e440b4d60 2024-03-25T17:30:54.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.317+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440706877 0 0) 0x7f0e441356e0 con 0x7f0e440b1730 2024-03-25T17:30:54.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.317+0000 7f0e3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:54.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.317+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:54.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.317+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:54.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.317+0000 7f0e3a7fc640 10 monclient(hunting): my global_id is 34268 2024-03-25T17:30:54.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.317+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e44135550 con 0x7f0e440b1730 2024-03-25T17:30:54.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.321+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13056560 0 0) 0x7f0e44140ae0 con 0x7f0e4414be90 2024-03-25T17:30:54.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.321+0000 7f0e3a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:30:54.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.321+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:30:54.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.321+0000 7f0e3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:30:54.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.321+0000 7f0e3a7fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T17:30:54.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.321+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e441356e0 con 0x7f0e4414be90 2024-03-25T17:30:54.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.338+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 377621846 0 0) 0x7f0e28004eb0 con 0x7f0e440b4d60 2024-03-25T17:30:54.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.338+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e44140ae0 con 0x7f0e440b4d60 2024-03-25T17:30:54.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.338+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0e34001910 con 0x7f0e440b4d60 2024-03-25T17:30:54.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.338+0000 7f0e3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:54.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.338+0000 7f0e3a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:30:54.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.338+0000 7f0e3a7fc640 10 monclient(hunting): dump: 2024-03-25T17:30:54.340 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:54.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.338+0000 7f0e3a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.343+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 389994151 0 0) 0x7f0e44135550 con 0x7f0e440b1730 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.343+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e28004eb0 con 0x7f0e440b1730 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.343+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0e40001970 con 0x7f0e440b1730 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.343+0000 7f0e3a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.343+0000 7f0e3a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.343+0000 7f0e3a7fc640 10 monclient(hunting): dump: 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.345 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:54.346 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:54.346 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:54.346 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:54.346 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:54.346 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:54.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.347+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2402587634 0 0) 0x7f0e44140ae0 con 0x7f0e440b4d60 2024-03-25T17:30:54.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.347+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 >> v1:172.21.15.134:6790/0 conn(0x7f0e440b1730 legacy=0x7f0e44134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.347+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 >> v1:172.21.15.134:6789/0 conn(0x7f0e4414be90 legacy=0x7f0e441db710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 1 monclient: found mon.b 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44135870 con 0x7f0e440b4d60 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 5 monclient: authenticate success, global_id 4291 2024-03-25T17:30:54.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 10 monclient: _renew_subs 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e44137a00 con 0x7f0e440b4d60 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 10 monclient: _renew_subs 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e44144bc0 con 0x7f0e440b4d60 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.348+0000 7f0e4bc40640 1 librados: init done 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.349+0000 7f0e4bc40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.349+0000 7f0e4bc40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:54.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.349+0000 7f0e4bc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:54.352 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.349+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e44144f10 con 0x7f0e440b4d60 2024-03-25T17:30:54.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.626+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e340047e0 con 0x7f0e440b4d60 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.626+0000 7f0e3a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.626+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0e34004b70 con 0x7f0e440b4d60 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.626+0000 7f0e3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.626+0000 7f0e3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.626+0000 7f0e3a7fc640 10 monclient: dump: 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:30:54.628 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:30:54.629 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:30:54.629 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:30:54.629 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:30:54.629 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.626+0000 7f0e3bfff640 4 set_mon_vals no callback set 2024-03-25T17:30:54.629 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.627+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f0e34004d50 con 0x7f0e440b4d60 2024-03-25T17:30:54.629 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.627+0000 7f0e499dc640 10 monclient: get_auth_request con 0x7f0e2806ded0 auth_method 0 2024-03-25T17:30:54.629 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.627+0000 7f0e3a7fc640 10 client.4291.objecter ms_handle_connect 0x7f0e2806ded0 2024-03-25T17:30:54.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.633+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f0e34054890 con 0x7f0e440b4d60 2024-03-25T17:30:54.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.633+0000 7f0e3a7fc640 10 client.4291.objecter ms_dispatch 0x7f0e44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:30:54.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.633+0000 7f0e3a7fc640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:30:54.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.633+0000 7f0e3a7fc640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:30:54.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.633+0000 7f0e3a7fc640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-25T17:30:54.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.634+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e340503a0 con 0x7f0e440b4d60 2024-03-25T17:30:54.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.634+0000 7f0e3a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:30:54.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.634+0000 7f0e3a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:30:54.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.878+0000 7f0e4bc40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:54.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.878+0000 7f0e4bc40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:54.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.878+0000 7f0e4bc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:30:54.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.878+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e440627b0 con 0x7f0e440b4d60 2024-03-25T17:30:54.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.973+0000 7f0e3a7fc640 1 -- 172.21.15.134:0/4251293306 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f0e340522d0 con 0x7f0e440b4d60 2024-03-25T17:30:54.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.974+0000 7f0e3a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:30:54.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.974+0000 7f0e3a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:30:54.975 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:30:54.975 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:30:54.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 10 librados: watch_flush enter 2024-03-25T17:30:54.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 10 librados: watch_flush exit 2024-03-25T17:30:54.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 10 client.4291.objecter successfully canceled tick 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 >> v1:172.21.15.134:6800/610837766 conn(0x7f0e2806ded0 legacy=0x7f0e28070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 10 monclient: shutdown 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 >> v1:172.21.15.159:6789/0 conn(0x7f0e440b4d60 legacy=0x7f0e441db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 shutdown_connections 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 >> 172.21.15.134:0/4251293306 conn(0x7f0e440617a0 msgr2=0x7f0e440b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 shutdown_connections 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 1 -- 172.21.15.134:0/4251293306 wait complete. 2024-03-25T17:30:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:30:54.976+0000 7f0e4bc40640 1 librados: shutdown 2024-03-25T17:30:54.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:01.795 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:01.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.912+0000 7fc4aebc4640 10 monclient: get_monmap_and_config 2024-03-25T17:31:01.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.912+0000 7fc4aebc4640 10 monclient: build_initial_monmap 2024-03-25T17:31:01.912 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.912+0000 7fc4aebc4640 10 monclient: monmap: 2024-03-25T17:31:01.912 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:01.912 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:01.912 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:01.913034+0000 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:01.913034+0000 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:01.913 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.913+0000 7fc4aebc4640 1 Processor -- start 2024-03-25T17:31:01.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.913+0000 7fc4aebc4640 1 -- start start 2024-03-25T17:31:01.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.913+0000 7fc4aebc4640 10 monclient: init 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.913+0000 7fc4aebc4640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.913+0000 7fc4aebc4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a806f360 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a806f740 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a8142e30 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8071d20 con 0x7fc4a806f740 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8071340 con 0x7fc4a806f360 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8071620 con 0x7fc4a8142e30 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:01.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.914+0000 7fc4aebc4640 10 monclient(hunting): authenticate will time out at 5415.177110s 2024-03-25T17:31:01.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.920+0000 7fc4ad161640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc4a8142e30 0x7fc4a814a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:48088/0 (socket says 172.21.15.134:48088) 2024-03-25T17:31:01.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.920+0000 7fc4ad161640 1 -- 172.21.15.134:0/1494423388 learned_addr learned my addr 172.21.15.134:0/1494423388 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:01.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.928+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628161277 0 0) 0x7fc4a8071620 con 0x7fc4a8142e30 2024-03-25T17:31:01.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.929+0000 7fc4a77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:01.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.929+0000 7fc4a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:01.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.929+0000 7fc4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:01.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.929+0000 7fc4a77fe640 10 monclient(hunting): my global_id is 34271 2024-03-25T17:31:01.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.929+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc498003520 con 0x7fc4a8142e30 2024-03-25T17:31:01.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.937+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663532877 0 0) 0x7fc4a8071d20 con 0x7fc4a806f740 2024-03-25T17:31:01.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.937+0000 7fc4a77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:01.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.937+0000 7fc4a77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:01.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.937+0000 7fc4a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:01.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.937+0000 7fc4a77fe640 10 monclient(hunting): my global_id is 4281 2024-03-25T17:31:01.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.937+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8071620 con 0x7fc4a806f740 2024-03-25T17:31:01.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 42272871 0 0) 0x7fc498003520 con 0x7fc4a8142e30 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 >> v1:172.21.15.159:6789/0 conn(0x7fc4a806f360 legacy=0x7fc4a8142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 >> v1:172.21.15.134:6789/0 conn(0x7fc4a806f740 legacy=0x7fc4a8072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 1 monclient: found mon.noname-b 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8071900 con 0x7fc4a8142e30 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 10 monclient: _finish_auth 0 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc4a0001a10 con 0x7fc4a8142e30 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:01.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4a77fe640 10 monclient: dump: 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4aebc4640 5 monclient: authenticate success, global_id 34271 2024-03-25T17:31:01.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.941+0000 7fc4aebc4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:01.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4a0001b70 con 0x7fc4a8142e30 2024-03-25T17:31:01.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4a77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:01.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4a77fe640 1 -- 172.21.15.134:0/1494423388 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc4a0001f00 con 0x7fc4a8142e30 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4aebc4640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4aebc4640 4 set_mon_vals no callback set 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4a77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.949+0000 7fc4a77fe640 10 monclient: dump: 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:01.950 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:01.951 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:01.951 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:01.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.950+0000 7fc4aebc4640 10 monclient: shutdown 2024-03-25T17:31:01.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.950+0000 7fc4aebc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:01.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.950+0000 7fc4aebc4640 1 -- 172.21.15.134:0/1494423388 >> v1:172.21.15.134:6790/0 conn(0x7fc4a8142e30 legacy=0x7fc4a814a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:01.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.950+0000 7fc4aebc4640 1 -- 172.21.15.134:0/1494423388 shutdown_connections 2024-03-25T17:31:01.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.950+0000 7fc4aebc4640 1 -- 172.21.15.134:0/1494423388 >> 172.21.15.134:0/1494423388 conn(0x7fc4a80b4b80 msgr2=0x7fc4a80b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:01.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.950+0000 7fc4aebc4640 1 -- 172.21.15.134:0/1494423388 shutdown_connections 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.950+0000 7fc4aebc4640 1 -- 172.21.15.134:0/1494423388 wait complete. 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 10 monclient: build_initial_monmap 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 10 monclient: monmap: 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:01.954 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 1 librados: starting msgr at 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 1 librados: starting objecter 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 1 Processor -- start 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 1 -- start start 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 1 librados: setting wanted keys 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 1 librados: calling monclient init 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.951+0000 7fc4aebc4640 10 monclient: init 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 10 monclient(hunting): picked mon.noname-c con 0x7fc4a806f360 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 10 monclient(hunting): picked mon.noname-b con 0x7fc4a806f740 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 10 monclient(hunting): picked mon.noname-a con 0x7fc4a8142e30 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:01.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a813b080 con 0x7fc4a8142e30 2024-03-25T17:31:01.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a81389b0 con 0x7fc4a806f740 2024-03-25T17:31:01.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a8138b40 con 0x7fc4a806f360 2024-03-25T17:31:01.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:01.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.952+0000 7fc4aebc4640 10 monclient(hunting): authenticate will time out at 5415.215301s 2024-03-25T17:31:01.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.969+0000 7fc4ad161640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc4a8142e30 0x7fc4a81e2160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:40650/0 (socket says 172.21.15.134:40650) 2024-03-25T17:31:01.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.969+0000 7fc4ad161640 1 -- 172.21.15.134:0/3929740105 learned_addr learned my addr 172.21.15.134:0/3929740105 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:01.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.971+0000 7fc4a57fa640 10 client.?.objecter ms_handle_connect 0x7fc4a8142e30 2024-03-25T17:31:01.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.971+0000 7fc4a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:01.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.972+0000 7fc4a57fa640 10 client.?.objecter ms_handle_connect 0x7fc4a806f740 2024-03-25T17:31:01.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.972+0000 7fc4a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:01.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.975+0000 7fc4a57fa640 10 client.?.objecter ms_handle_connect 0x7fc4a806f360 2024-03-25T17:31:01.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.975+0000 7fc4a57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:01.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.976+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807961079 0 0) 0x7fc4a813b080 con 0x7fc4a8142e30 2024-03-25T17:31:01.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.976+0000 7fc4a57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:01.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.976+0000 7fc4a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:01.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.976+0000 7fc4a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:01.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.976+0000 7fc4a57fa640 10 monclient(hunting): my global_id is 4284 2024-03-25T17:31:01.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.976+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc48c004eb0 con 0x7fc4a8142e30 2024-03-25T17:31:01.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.984+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982154307 0 0) 0x7fc4a8138b40 con 0x7fc4a806f360 2024-03-25T17:31:01.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.984+0000 7fc4a57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:01.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.984+0000 7fc4a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:01.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.984+0000 7fc4a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:01.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.984+0000 7fc4a57fa640 10 monclient(hunting): my global_id is 34274 2024-03-25T17:31:01.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.984+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a813b080 con 0x7fc4a806f360 2024-03-25T17:31:01.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.985+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1109252938 0 0) 0x7fc4a81389b0 con 0x7fc4a806f740 2024-03-25T17:31:01.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.985+0000 7fc4a57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:01.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.985+0000 7fc4a57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:01.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.985+0000 7fc4a57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:01.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.985+0000 7fc4a57fa640 10 monclient(hunting): my global_id is 4297 2024-03-25T17:31:01.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.985+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4a8138b40 con 0x7fc4a806f740 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.995+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1057324916 0 0) 0x7fc48c004eb0 con 0x7fc4a8142e30 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.995+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc4a81389b0 con 0x7fc4a8142e30 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.995+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc4a0001e80 con 0x7fc4a8142e30 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.995+0000 7fc4a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.995+0000 7fc4a57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.995+0000 7fc4a57fa640 10 monclient(hunting): dump: 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:01.996 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:01.997 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:01.997 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:01.997 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:01.997 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:01.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.995+0000 7fc4a57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:01.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.999+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 893508955 0 0) 0x7fc4a813b080 con 0x7fc4a806f360 2024-03-25T17:31:01.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.999+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc48c004eb0 con 0x7fc4a806f360 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.999+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc49c0018b0 con 0x7fc4a806f360 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.999+0000 7fc4a57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.999+0000 7fc4a57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:01.999+0000 7fc4a57fa640 10 monclient(hunting): dump: 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:02.000 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:02.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 553495853 0 0) 0x7fc4a81389b0 con 0x7fc4a8142e30 2024-03-25T17:31:02.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 >> v1:172.21.15.134:6790/0 conn(0x7fc4a806f360 legacy=0x7fc4a8139930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:02.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 >> v1:172.21.15.159:6789/0 conn(0x7fc4a806f740 legacy=0x7fc4a81dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:02.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 10 monclient: _finish_hunting 0 2024-03-25T17:31:02.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 1 monclient: found mon.a 2024-03-25T17:31:02.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a8138cd0 con 0x7fc4a8142e30 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 10 monclient: _finish_auth 0 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 10 monclient: _check_auth_tickets 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 5 monclient: authenticate success, global_id 4284 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 10 monclient: _renew_subs 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a813fc40 con 0x7fc4a8142e30 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 10 monclient: _renew_subs 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a813b4a0 con 0x7fc4a8142e30 2024-03-25T17:31:02.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.005+0000 7fc4aebc4640 1 librados: init done 2024-03-25T17:31:02.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.006+0000 7fc4aebc4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:02.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.006+0000 7fc4aebc4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:02.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.007+0000 7fc4aebc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:02.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.007+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a80627b0 con 0x7fc4a8142e30 2024-03-25T17:31:02.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.316+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc4a00015b0 con 0x7fc4a8142e30 2024-03-25T17:31:02.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.316+0000 7fc4a57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:02.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.316+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc4a00020c0 con 0x7fc4a8142e30 2024-03-25T17:31:02.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.316+0000 7fc4a57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:02.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.316+0000 7fc4a57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:02.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.316+0000 7fc4a57fa640 10 monclient: dump: 2024-03-25T17:31:02.317 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:02.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.316+0000 7fc4a6ffd640 4 set_mon_vals no callback set 2024-03-25T17:31:02.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.317+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fc4a001ad00 con 0x7fc4a8142e30 2024-03-25T17:31:02.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.318+0000 7fc4ac960640 10 monclient: get_auth_request con 0x7fc48c06df70 auth_method 0 2024-03-25T17:31:02.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.318+0000 7fc4a57fa640 10 client.4284.objecter ms_handle_connect 0x7fc48c06df70 2024-03-25T17:31:02.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fc4a00549b0 con 0x7fc4a8142e30 2024-03-25T17:31:02.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 10 client.4284.objecter ms_dispatch 0x7fc4a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:02.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-25T17:31:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc4a00502f0 con 0x7fc4a8142e30 2024-03-25T17:31:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.324+0000 7fc4a57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:02.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.568+0000 7fc4aebc4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:02.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.568+0000 7fc4aebc4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:02.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.568+0000 7fc4aebc4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:02.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.568+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4a8139530 con 0x7fc4a8142e30 2024-03-25T17:31:02.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.617+0000 7fc4a57fa640 1 -- 172.21.15.134:0/3929740105 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fc4a00523f0 con 0x7fc4a8142e30 2024-03-25T17:31:02.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.617+0000 7fc4a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:02.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.617+0000 7fc4a57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:02.618 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:02.618 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:02.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 10 librados: watch_flush enter 2024-03-25T17:31:02.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 10 librados: watch_flush exit 2024-03-25T17:31:02.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 10 client.4284.objecter successfully canceled tick 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 >> v1:172.21.15.134:6800/610837766 conn(0x7fc48c06df70 legacy=0x7fc48c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 10 monclient: shutdown 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 >> v1:172.21.15.134:6789/0 conn(0x7fc4a8142e30 legacy=0x7fc4a81e2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 shutdown_connections 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 >> 172.21.15.134:0/3929740105 conn(0x7fc4a80b4b80 msgr2=0x7fc4a813c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 shutdown_connections 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 1 -- 172.21.15.134:0/3929740105 wait complete. 2024-03-25T17:31:02.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:02.620+0000 7fc4aebc4640 1 librados: shutdown 2024-03-25T17:31:02.634 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:04.249 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:09.265 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.251 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:09.268 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.290 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:09.306 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.296 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:09.325 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.298 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:09.314 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.302 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:09.331 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.321 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:09.338 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.346 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:09.375 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:04.360 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:09.377 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:09.440 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.558+0000 7f1ebed23640 10 monclient: get_monmap_and_config 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.558+0000 7f1ebed23640 10 monclient: build_initial_monmap 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.558+0000 7f1ebed23640 10 monclient: monmap: 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:09.559543+0000 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:09.559543+0000 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:09.559 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:09.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.559+0000 7f1ebed23640 1 Processor -- start 2024-03-25T17:31:09.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.559+0000 7f1ebed23640 1 -- start start 2024-03-25T17:31:09.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.559+0000 7f1ebed23640 10 monclient: init 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8142520 con 0x7f1eb80a98f0 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80b28a0 con 0x7f1eb81430c0 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80b2a30 con 0x7f1eb813f710 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:09.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.560+0000 7f1ebed23640 10 monclient(hunting): authenticate will time out at 5422.823066s 2024-03-25T17:31:09.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.571+0000 7f1eb7fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f1eb813f710 0x7f1eb81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:39594/0 (socket says 172.21.15.134:39594) 2024-03-25T17:31:09.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.571+0000 7f1eb7fff640 1 -- 172.21.15.134:0/3067071208 learned_addr learned my addr 172.21.15.134:0/3067071208 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:09.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.587+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011129506 0 0) 0x7f1eb80b28a0 con 0x7f1eb81430c0 2024-03-25T17:31:09.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.587+0000 7f1eb77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:09.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.587+0000 7f1eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:09.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.587+0000 7f1eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:09.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.587+0000 7f1eb77fe640 10 monclient(hunting): my global_id is 4300 2024-03-25T17:31:09.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.587+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8003520 con 0x7f1eb81430c0 2024-03-25T17:31:09.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.591+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382597938 0 0) 0x7f1eb80b2a30 con 0x7f1eb813f710 2024-03-25T17:31:09.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.591+0000 7f1eb77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:09.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.591+0000 7f1eb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:09.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.591+0000 7f1eb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:09.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.591+0000 7f1eb77fe640 10 monclient(hunting): my global_id is 34277 2024-03-25T17:31:09.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.591+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb80b28a0 con 0x7f1eb813f710 2024-03-25T17:31:09.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2580887046 0 0) 0x7f1eb80b28a0 con 0x7f1eb813f710 2024-03-25T17:31:09.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 >> v1:172.21.15.159:6789/0 conn(0x7f1eb81430c0 legacy=0x7f1eb814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 >> v1:172.21.15.134:6789/0 conn(0x7f1eb80a98f0 legacy=0x7f1eb80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 1 monclient: found mon.noname-b 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb80b44c0 con 0x7f1eb813f710 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 10 monclient: _finish_auth 0 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1ea4001670 con 0x7f1eb813f710 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1eb77fe640 10 monclient: dump: 2024-03-25T17:31:09.601 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1ebed23640 5 monclient: authenticate success, global_id 34277 2024-03-25T17:31:09.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.600+0000 7f1ebed23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:09.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ea40017d0 con 0x7f1eb813f710 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1eb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1eb77fe640 1 -- 172.21.15.134:0/3067071208 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1ea4001ba0 con 0x7f1eb813f710 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1ebed23640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1ebed23640 4 set_mon_vals no callback set 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1eb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1eb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.611+0000 7f1eb77fe640 10 monclient: dump: 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:09.612 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:09.613 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:09.613 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:09.613 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:09.613 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:09.613 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:09.613 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.612+0000 7f1ebed23640 10 monclient: shutdown 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.612+0000 7f1ebed23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.612+0000 7f1ebed23640 1 -- 172.21.15.134:0/3067071208 >> v1:172.21.15.134:6790/0 conn(0x7f1eb813f710 legacy=0x7f1eb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.612+0000 7f1ebed23640 1 -- 172.21.15.134:0/3067071208 shutdown_connections 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.612+0000 7f1ebed23640 1 -- 172.21.15.134:0/3067071208 >> 172.21.15.134:0/3067071208 conn(0x7f1eb80b7ea0 msgr2=0x7f1eb80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.612+0000 7f1ebed23640 1 -- 172.21.15.134:0/3067071208 shutdown_connections 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.612+0000 7f1ebed23640 1 -- 172.21.15.134:0/3067071208 wait complete. 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 10 monclient: build_initial_monmap 2024-03-25T17:31:09.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 10 monclient: monmap: 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 1 librados: starting msgr at 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 1 librados: starting objecter 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 1 Processor -- start 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 1 -- start start 2024-03-25T17:31:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 1 librados: setting wanted keys 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 1 librados: calling monclient init 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.613+0000 7f1ebed23640 10 monclient: init 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 10 monclient(hunting): picked mon.noname-a con 0x7f1eb80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 10 monclient(hunting): picked mon.noname-c con 0x7f1eb813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 10 monclient(hunting): picked mon.noname-b con 0x7f1eb81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb8139740 con 0x7f1eb80a98f0 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb81d4c00 con 0x7f1eb81430c0 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb80b16d0 con 0x7f1eb813f710 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:09.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.614+0000 7f1ebed23640 10 monclient(hunting): authenticate will time out at 5422.877580s 2024-03-25T17:31:09.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.620+0000 7f1eb7fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f1eb813f710 0x7f1eb80afe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:39648/0 (socket says 172.21.15.134:39648) 2024-03-25T17:31:09.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.620+0000 7f1eb7fff640 1 -- 172.21.15.134:0/3827498420 learned_addr learned my addr 172.21.15.134:0/3827498420 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:09.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.623+0000 7f1eb57fa640 10 client.?.objecter ms_handle_connect 0x7f1eb813f710 2024-03-25T17:31:09.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.623+0000 7f1eb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:09.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.624+0000 7f1eb57fa640 10 client.?.objecter ms_handle_connect 0x7f1eb81430c0 2024-03-25T17:31:09.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.624+0000 7f1eb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:09.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.626+0000 7f1eb57fa640 10 client.?.objecter ms_handle_connect 0x7f1eb80a98f0 2024-03-25T17:31:09.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.626+0000 7f1eb57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:09.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.638+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 631283815 0 0) 0x7f1eb80b16d0 con 0x7f1eb813f710 2024-03-25T17:31:09.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.638+0000 7f1eb57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:09.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.638+0000 7f1eb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:09.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.638+0000 7f1eb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:09.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.638+0000 7f1eb57fa640 10 monclient(hunting): my global_id is 34280 2024-03-25T17:31:09.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.638+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e9c004eb0 con 0x7f1eb813f710 2024-03-25T17:31:09.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.652+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576034984 0 0) 0x7f1eb8139740 con 0x7f1eb80a98f0 2024-03-25T17:31:09.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.652+0000 7f1eb57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:09.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.652+0000 7f1eb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:09.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.652+0000 7f1eb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:09.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.652+0000 7f1eb57fa640 10 monclient(hunting): my global_id is 4290 2024-03-25T17:31:09.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.652+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb80b16d0 con 0x7f1eb80a98f0 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.675+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3607725934 0 0) 0x7f1eb80b16d0 con 0x7f1eb80a98f0 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.675+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eb8139740 con 0x7f1eb80a98f0 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.675+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1eac0018a0 con 0x7f1eb80a98f0 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.675+0000 7f1eb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.675+0000 7f1eb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.675+0000 7f1eb57fa640 10 monclient(hunting): dump: 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:09.676 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:09.677 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:09.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.675+0000 7f1eb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:09.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.743+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285564912 0 0) 0x7f1eb81d4c00 con 0x7f1eb81430c0 2024-03-25T17:31:09.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.743+0000 7f1eb57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:09.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.743+0000 7f1eb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:09.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.743+0000 7f1eb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:09.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.743+0000 7f1eb57fa640 10 monclient(hunting): my global_id is 4303 2024-03-25T17:31:09.743 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.743+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb80b16d0 con 0x7f1eb81430c0 2024-03-25T17:31:09.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1238445956 0 0) 0x7f1eb80b16d0 con 0x7f1eb81430c0 2024-03-25T17:31:09.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1eb81d4c00 con 0x7f1eb81430c0 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3612046553 0 0) 0x7f1eb8139740 con 0x7f1eb80a98f0 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 >> v1:172.21.15.134:6790/0 conn(0x7f1eb813f710 legacy=0x7f1eb80afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 >> v1:172.21.15.159:6789/0 conn(0x7f1eb81430c0 legacy=0x7f1eb81d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 10 monclient: _finish_hunting 0 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 1 monclient: found mon.a 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb80b0c30 con 0x7f1eb80a98f0 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 10 monclient: _finish_auth 0 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 10 monclient: _check_auth_tickets 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1eb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 5 monclient: authenticate success, global_id 4290 2024-03-25T17:31:09.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 10 monclient: _renew_subs 2024-03-25T17:31:09.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:09.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb80b0ee0 con 0x7f1eb80a98f0 2024-03-25T17:31:09.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:09.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 10 monclient: _renew_subs 2024-03-25T17:31:09.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:09.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb80b0780 con 0x7f1eb80a98f0 2024-03-25T17:31:09.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.754+0000 7f1ebed23640 1 librados: init done 2024-03-25T17:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.756+0000 7f1ebed23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.756+0000 7f1ebed23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.756+0000 7f1ebed23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:09.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:09.756+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb80b0a80 con 0x7f1eb80a98f0 2024-03-25T17:31:10.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.070+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1eac001d50 con 0x7f1eb80a98f0 2024-03-25T17:31:10.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.070+0000 7f1eb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:10.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.070+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1eac003790 con 0x7f1eb80a98f0 2024-03-25T17:31:10.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.070+0000 7f1eb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.070+0000 7f1eb57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.070+0000 7f1eb57fa640 10 monclient: dump: 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:10.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.070+0000 7f1eb6ffd640 4 set_mon_vals no callback set 2024-03-25T17:31:10.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.071+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f1eac0039b0 con 0x7f1eb80a98f0 2024-03-25T17:31:10.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.072+0000 7f1eb7fff640 10 monclient: get_auth_request con 0x7f1e9c06e0e0 auth_method 0 2024-03-25T17:31:10.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.072+0000 7f1eb57fa640 10 client.4290.objecter ms_handle_connect 0x7f1e9c06e0e0 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f1eac053570 con 0x7f1eb80a98f0 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 10 client.4290.objecter ms_dispatch 0x7f1eb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1eac04eeb0 con 0x7f1eb80a98f0 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:10.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.078+0000 7f1eb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:10.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.330+0000 7f1ebed23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:10.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.330+0000 7f1ebed23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:10.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.330+0000 7f1ebed23640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:10.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.330+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1eb80b3ef0 con 0x7f1eb80a98f0 2024-03-25T17:31:10.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.388+0000 7f1eb57fa640 1 -- 172.21.15.134:0/3827498420 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f1eac050fb0 con 0x7f1eb80a98f0 2024-03-25T17:31:10.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.388+0000 7f1eb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:10.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.388+0000 7f1eb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:10.389 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:10.389 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.390+0000 7f1ebed23640 10 librados: watch_flush enter 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 10 librados: watch_flush exit 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 10 client.4290.objecter successfully canceled tick 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 >> v1:172.21.15.134:6800/610837766 conn(0x7f1e9c06e0e0 legacy=0x7f1e9c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 10 monclient: shutdown 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 >> v1:172.21.15.134:6789/0 conn(0x7f1eb80a98f0 legacy=0x7f1eb81d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 shutdown_connections 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 >> 172.21.15.134:0/3827498420 conn(0x7f1eb80b7ea0 msgr2=0x7f1eb8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 shutdown_connections 2024-03-25T17:31:10.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 1 -- 172.21.15.134:0/3827498420 wait complete. 2024-03-25T17:31:10.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:10.391+0000 7f1ebed23640 1 librados: shutdown 2024-03-25T17:31:10.404 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:17.210 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.325+0000 7f5fffe50640 10 monclient: get_monmap_and_config 2024-03-25T17:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.325+0000 7f5fffe50640 10 monclient: build_initial_monmap 2024-03-25T17:31:17.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.325+0000 7f5fffe50640 10 monclient: monmap: 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:17.326124+0000 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:17.326124+0000 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:17.326 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.325+0000 7f5fffe50640 1 Processor -- start 2024-03-25T17:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.326+0000 7f5fffe50640 1 -- start start 2024-03-25T17:31:17.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.326+0000 7f5fffe50640 10 monclient: init 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.326+0000 7f5fffe50640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.326+0000 7f5fffe50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.326+0000 7f5fffe50640 10 monclient(hunting): picked mon.noname-b con 0x7f5ff80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.326+0000 7f5fffe50640 10 monclient(hunting): picked mon.noname-c con 0x7f5ff813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.327+0000 7f5fffe50640 10 monclient(hunting): picked mon.noname-a con 0x7f5ff81430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.327+0000 7f5fffe50640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff8142520 con 0x7f5ff81430c0 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.327+0000 7f5fffe50640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff80b28a0 con 0x7f5ff813f710 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.327+0000 7f5fffe50640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff80b2a30 con 0x7f5ff80a98f0 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.327+0000 7f5fffe50640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:17.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.327+0000 7f5fffe50640 10 monclient(hunting): authenticate will time out at 5430.589882s 2024-03-25T17:31:17.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.338+0000 7f5ffd3eb640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f5ff813f710 0x7f5ff81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54364/0 (socket says 172.21.15.134:54364) 2024-03-25T17:31:17.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.338+0000 7f5ffd3eb640 1 -- 172.21.15.134:0/896038479 learned_addr learned my addr 172.21.15.134:0/896038479 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:17.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.347+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3401128368 0 0) 0x7f5ff80b2a30 con 0x7f5ff80a98f0 2024-03-25T17:31:17.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.347+0000 7f5ffcbea640 10 monclient(hunting): _negotiate 2024-03-25T17:31:17.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.347+0000 7f5ffcbea640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:17.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.347+0000 7f5ffcbea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:17.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.347+0000 7f5ffcbea640 10 monclient(hunting): my global_id is 34283 2024-03-25T17:31:17.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.347+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe4003520 con 0x7f5ff80a98f0 2024-03-25T17:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.348+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357474854 0 0) 0x7f5ff80b28a0 con 0x7f5ff813f710 2024-03-25T17:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.348+0000 7f5ffcbea640 10 monclient(hunting): _negotiate 2024-03-25T17:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.348+0000 7f5ffcbea640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.348+0000 7f5ffcbea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.348+0000 7f5ffcbea640 10 monclient(hunting): my global_id is 4306 2024-03-25T17:31:17.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.348+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff80b2a30 con 0x7f5ff813f710 2024-03-25T17:31:17.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.355+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580510477 0 0) 0x7f5ff8142520 con 0x7f5ff81430c0 2024-03-25T17:31:17.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.355+0000 7f5ffcbea640 10 monclient(hunting): _negotiate 2024-03-25T17:31:17.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.355+0000 7f5ffcbea640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:17.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.355+0000 7f5ffcbea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:17.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.355+0000 7f5ffcbea640 10 monclient(hunting): my global_id is 4293 2024-03-25T17:31:17.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.355+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff80b28a0 con 0x7f5ff81430c0 2024-03-25T17:31:17.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.356+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108162609 0 0) 0x7f5fe4003520 con 0x7f5ff80a98f0 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 >> v1:172.21.15.159:6789/0 conn(0x7f5ff813f710 legacy=0x7f5ff81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 >> v1:172.21.15.134:6789/0 conn(0x7f5ff81430c0 legacy=0x7f5ff814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 10 monclient: _finish_hunting 0 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 1 monclient: found mon.noname-b 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff80b44c0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 10 monclient: _finish_auth 0 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 10 monclient: _check_auth_tickets 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5ff4001690 con 0x7f5ff80a98f0 2024-03-25T17:31:17.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5fffe50640 5 monclient: authenticate success, global_id 34283 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5fffe50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5ffcbea640 10 monclient: dump: 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:17.359 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:17.360 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:17.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.357+0000 7f5fffe50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:17.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff40017f0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5ffcbea640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5ffcbea640 1 -- 172.21.15.134:0/896038479 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5ff4001c00 con 0x7f5ff80a98f0 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5fffe50640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5fffe50640 4 set_mon_vals no callback set 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5ffcbea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5ffcbea640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.378+0000 7f5ffcbea640 10 monclient: dump: 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:17.379 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:17.380 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:17.380 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:17.380 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:17.380 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.379+0000 7f5fffe50640 10 monclient: shutdown 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.379+0000 7f5fffe50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.379+0000 7f5fffe50640 1 -- 172.21.15.134:0/896038479 >> v1:172.21.15.134:6790/0 conn(0x7f5ff80a98f0 legacy=0x7f5ff80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.379+0000 7f5fffe50640 1 -- 172.21.15.134:0/896038479 shutdown_connections 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.379+0000 7f5fffe50640 1 -- 172.21.15.134:0/896038479 >> 172.21.15.134:0/896038479 conn(0x7f5ff80b7ea0 msgr2=0x7f5ff80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.379+0000 7f5fffe50640 1 -- 172.21.15.134:0/896038479 shutdown_connections 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.379+0000 7f5fffe50640 1 -- 172.21.15.134:0/896038479 wait complete. 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 10 monclient: build_initial_monmap 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 10 monclient: monmap: 2024-03-25T17:31:17.381 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 1 librados: starting msgr at 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 1 librados: starting objecter 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 1 Processor -- start 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 1 -- start start 2024-03-25T17:31:17.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 1 librados: setting wanted keys 2024-03-25T17:31:17.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 1 librados: calling monclient init 2024-03-25T17:31:17.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.380+0000 7f5fffe50640 10 monclient: init 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 10 monclient(hunting): picked mon.noname-b con 0x7f5ff80a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 10 monclient(hunting): picked mon.noname-a con 0x7f5ff813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 10 monclient(hunting): picked mon.noname-c con 0x7f5ff81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff8070490 con 0x7f5ff813f710 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff806e0a0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff806e230 con 0x7f5ff81430c0 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:17.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.381+0000 7f5fffe50640 10 monclient(hunting): authenticate will time out at 5430.644364s 2024-03-25T17:31:17.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.397+0000 7f5ffdbec640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f5ff80a98f0 0x7f5ff81df440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54416/0 (socket says 172.21.15.134:54416) 2024-03-25T17:31:17.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.397+0000 7f5ffdbec640 1 -- 172.21.15.134:0/3915924876 learned_addr learned my addr 172.21.15.134:0/3915924876 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:17.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.397+0000 7f5fee7fc640 10 client.?.objecter ms_handle_connect 0x7f5ff80a98f0 2024-03-25T17:31:17.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.397+0000 7f5fee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:17.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.403+0000 7f5fee7fc640 10 client.?.objecter ms_handle_connect 0x7f5ff813f710 2024-03-25T17:31:17.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.403+0000 7f5fee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 10 client.?.objecter ms_handle_connect 0x7f5ff81430c0 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057557815 0 0) 0x7f5ff806e0a0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T17:31:17.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.406+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fdc004eb0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.419+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987236030 0 0) 0x7f5ff806e230 con 0x7f5ff81430c0 2024-03-25T17:31:17.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.419+0000 7f5fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:31:17.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.419+0000 7f5fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:17.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.419+0000 7f5fee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:17.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.419+0000 7f5fee7fc640 10 monclient(hunting): my global_id is 34286 2024-03-25T17:31:17.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.419+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff806e0a0 con 0x7f5ff81430c0 2024-03-25T17:31:17.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.420+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572103778 0 0) 0x7f5ff8070490 con 0x7f5ff813f710 2024-03-25T17:31:17.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.420+0000 7f5fee7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:31:17.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.420+0000 7f5fee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:17.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.420+0000 7f5fee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:17.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.421+0000 7f5fee7fc640 10 monclient(hunting): my global_id is 4296 2024-03-25T17:31:17.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.421+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff806e230 con 0x7f5ff813f710 2024-03-25T17:31:17.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.425+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3570379053 0 0) 0x7f5fdc004eb0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.425+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ff8070490 con 0x7f5ff80a98f0 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.425+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5ff4001bc0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.425+0000 7f5fee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.425+0000 7f5fee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.425+0000 7f5fee7fc640 10 monclient(hunting): dump: 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:17.426 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:17.427 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:17.427 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:17.427 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.425+0000 7f5fee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3662183054 0 0) 0x7f5ff8070490 con 0x7f5ff80a98f0 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 >> v1:172.21.15.134:6790/0 conn(0x7f5ff81430c0 legacy=0x7f5ff81e64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 >> v1:172.21.15.134:6789/0 conn(0x7f5ff813f710 legacy=0x7f5ff806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 1 monclient: found mon.b 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff8070b20 con 0x7f5ff80a98f0 2024-03-25T17:31:17.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 10 monclient: _finish_auth 0 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 5 monclient: authenticate success, global_id 4309 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 10 monclient: _renew_subs 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff81dfb50 con 0x7f5ff80a98f0 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 10 monclient: _renew_subs 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff806f580 con 0x7f5ff80a98f0 2024-03-25T17:31:17.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.431+0000 7f5fffe50640 1 librados: init done 2024-03-25T17:31:17.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.432+0000 7f5fffe50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:17.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.433+0000 7f5fffe50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:17.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.433+0000 7f5fffe50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:17.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.433+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff8070d70 con 0x7f5ff80a98f0 2024-03-25T17:31:17.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.451+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff40011f0 con 0x7f5ff80a98f0 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.451+0000 7f5fee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.451+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5ff4003b00 con 0x7f5ff80a98f0 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.451+0000 7f5fee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.451+0000 7f5fee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.451+0000 7f5fee7fc640 10 monclient: dump: 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:17.452 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:17.453 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:17.453 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:17.453 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:17.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.451+0000 7f5feffff640 4 set_mon_vals no callback set 2024-03-25T17:31:17.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.452+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f5ff4003d20 con 0x7f5ff80a98f0 2024-03-25T17:31:17.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.452+0000 7f5ffd3eb640 10 monclient: get_auth_request con 0x7f5fdc06ddf0 auth_method 0 2024-03-25T17:31:17.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.453+0000 7f5fee7fc640 10 client.4309.objecter ms_handle_connect 0x7f5fdc06ddf0 2024-03-25T17:31:17.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.454+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f5ff4054880 con 0x7f5ff80a98f0 2024-03-25T17:31:17.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.454+0000 7f5fee7fc640 10 client.4309.objecter ms_dispatch 0x7f5ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:17.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.454+0000 7f5fee7fc640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:17.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.454+0000 7f5fee7fc640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:17.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.454+0000 7f5fee7fc640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-25T17:31:17.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.717+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ff4050390 con 0x7f5ff80a98f0 2024-03-25T17:31:17.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.717+0000 7f5fee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:17.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.717+0000 7f5fee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:17.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.964+0000 7f5fffe50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:17.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.964+0000 7f5fffe50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:17.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.964+0000 7f5fffe50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:17.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:17.964+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ff81df080 con 0x7f5ff80a98f0 2024-03-25T17:31:18.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.000+0000 7f5fee7fc640 1 -- 172.21.15.134:0/3915924876 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f5ff40522c0 con 0x7f5ff80a98f0 2024-03-25T17:31:18.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.000+0000 7f5fee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:18.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.000+0000 7f5fee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:18.001 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:18.001 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.002+0000 7f5fffe50640 10 librados: watch_flush enter 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 10 librados: watch_flush exit 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 10 client.4309.objecter successfully canceled tick 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 >> v1:172.21.15.134:6800/610837766 conn(0x7f5fdc06ddf0 legacy=0x7f5fdc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 10 monclient: shutdown 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 >> v1:172.21.15.159:6789/0 conn(0x7f5ff80a98f0 legacy=0x7f5ff81df440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 shutdown_connections 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 >> 172.21.15.134:0/3915924876 conn(0x7f5ff80b7ea0 msgr2=0x7f5ff8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 shutdown_connections 2024-03-25T17:31:18.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 1 -- 172.21.15.134:0/3915924876 wait complete. 2024-03-25T17:31:18.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:18.003+0000 7f5fffe50640 1 librados: shutdown 2024-03-25T17:31:18.014 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:24.320 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:29.336 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.323 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:29.339 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.364 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:29.380 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.373 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:29.390 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.380 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:29.397 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.393 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:29.422 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.400 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:29.429 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.424 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:29.441 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.432 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:29.460 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:24.820 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.933+0000 7f699c4f4640 10 monclient: get_monmap_and_config 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.933+0000 7f699c4f4640 10 monclient: build_initial_monmap 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.933+0000 7f699c4f4640 10 monclient: monmap: 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:24.934747+0000 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:24.934747+0000 2024-03-25T17:31:24.934 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:24.935 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:24.935 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:24.935 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:24.935 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:24.935 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.934+0000 7f699c4f4640 1 Processor -- start 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.934+0000 7f699c4f4640 1 -- start start 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.934+0000 7f699c4f4640 10 monclient: init 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 10 monclient(hunting): picked mon.noname-a con 0x7f69940a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 10 monclient(hunting): picked mon.noname-c con 0x7f699413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 10 monclient(hunting): picked mon.noname-b con 0x7f69941430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:24.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6994142520 con 0x7f69940a98f0 2024-03-25T17:31:24.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69940b28a0 con 0x7f699413f710 2024-03-25T17:31:24.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69940b2a30 con 0x7f69941430c0 2024-03-25T17:31:24.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:24.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.935+0000 7f699c4f4640 10 monclient(hunting): authenticate will time out at 5438.198361s 2024-03-25T17:31:24.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.943+0000 7f699a290640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f69940a98f0 0x7f69940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:43392/0 (socket says 172.21.15.134:43392) 2024-03-25T17:31:24.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.943+0000 7f699a290640 1 -- 172.21.15.134:0/1087288025 learned_addr learned my addr 172.21.15.134:0/1087288025 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:24.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.955+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072601146 0 0) 0x7f69940b28a0 con 0x7f699413f710 2024-03-25T17:31:24.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.955+0000 7f699928e640 10 monclient(hunting): _negotiate 2024-03-25T17:31:24.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.955+0000 7f699928e640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:24.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.955+0000 7f699928e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:24.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.955+0000 7f699928e640 10 monclient(hunting): my global_id is 4312 2024-03-25T17:31:24.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.955+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6980003520 con 0x7f699413f710 2024-03-25T17:31:24.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.957+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245842673 0 0) 0x7f69940b2a30 con 0x7f69941430c0 2024-03-25T17:31:24.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.957+0000 7f699928e640 10 monclient(hunting): _negotiate 2024-03-25T17:31:24.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.957+0000 7f699928e640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:24.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.957+0000 7f699928e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:24.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.957+0000 7f699928e640 10 monclient(hunting): my global_id is 34289 2024-03-25T17:31:24.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.957+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69940b28a0 con 0x7f69941430c0 2024-03-25T17:31:24.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.960+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241366650 0 0) 0x7f6994142520 con 0x7f69940a98f0 2024-03-25T17:31:24.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.960+0000 7f699928e640 10 monclient(hunting): _negotiate 2024-03-25T17:31:24.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.960+0000 7f699928e640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:24.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.960+0000 7f699928e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:24.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.960+0000 7f699928e640 10 monclient(hunting): my global_id is 4299 2024-03-25T17:31:24.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.960+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69940b2a30 con 0x7f69940a98f0 2024-03-25T17:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2166230880 0 0) 0x7f6980003520 con 0x7f699413f710 2024-03-25T17:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 >> v1:172.21.15.134:6790/0 conn(0x7f69941430c0 legacy=0x7f699414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 >> v1:172.21.15.134:6789/0 conn(0x7f69940a98f0 legacy=0x7f69940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 10 monclient: _finish_hunting 0 2024-03-25T17:31:24.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 1 monclient: found mon.noname-c 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69940b44c0 con 0x7f699413f710 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 10 monclient: _finish_auth 0 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 10 monclient: _check_auth_tickets 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.966+0000 7f699928e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.967+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6984001690 con 0x7f699413f710 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.967+0000 7f699928e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.967+0000 7f699928e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.967+0000 7f699928e640 10 monclient: dump: 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:24.968 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:24.969 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:24.969 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:24.969 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:24.969 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:24.969 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:24.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.967+0000 7f699c4f4640 5 monclient: authenticate success, global_id 4312 2024-03-25T17:31:24.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.967+0000 7f699c4f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69840017f0 con 0x7f699413f710 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699928e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699928e640 1 -- 172.21.15.134:0/1087288025 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6984001c00 con 0x7f699413f710 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699928e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699928e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699928e640 10 monclient: dump: 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:24.992 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:24.993 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:24.993 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:24.993 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:24.993 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:24.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699c4f4640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:24.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.991+0000 7f699c4f4640 4 set_mon_vals no callback set 2024-03-25T17:31:24.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.992+0000 7f699c4f4640 10 monclient: shutdown 2024-03-25T17:31:24.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.992+0000 7f699c4f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:24.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.992+0000 7f699c4f4640 1 -- 172.21.15.134:0/1087288025 >> v1:172.21.15.159:6789/0 conn(0x7f699413f710 legacy=0x7f69941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:24.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.992+0000 7f699c4f4640 1 -- 172.21.15.134:0/1087288025 shutdown_connections 2024-03-25T17:31:24.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.992+0000 7f699c4f4640 1 -- 172.21.15.134:0/1087288025 >> 172.21.15.134:0/1087288025 conn(0x7f69940b7ea0 msgr2=0x7f69940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:24.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.992+0000 7f699c4f4640 1 -- 172.21.15.134:0/1087288025 shutdown_connections 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.992+0000 7f699c4f4640 1 -- 172.21.15.134:0/1087288025 wait complete. 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 10 monclient: build_initial_monmap 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 10 monclient: monmap: 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:24.996 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 1 librados: starting msgr at 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 1 librados: starting objecter 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 1 Processor -- start 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 1 -- start start 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 1 librados: setting wanted keys 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 1 librados: calling monclient init 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.993+0000 7f699c4f4640 10 monclient: init 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 10 monclient(hunting): picked mon.noname-a con 0x7f69940a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 10 monclient(hunting): picked mon.noname-b con 0x7f699413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 10 monclient(hunting): picked mon.noname-c con 0x7f69941430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69940abdc0 con 0x7f69940a98f0 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69941d7730 con 0x7f699413f710 2024-03-25T17:31:24.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6994138f10 con 0x7f69941430c0 2024-03-25T17:31:24.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:24.998 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.994+0000 7f699c4f4640 10 monclient(hunting): authenticate will time out at 5438.257408s 2024-03-25T17:31:24.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.999+0000 7f6999a8f640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f699413f710 0x7f69941d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:53084/0 (socket says 172.21.15.134:53084) 2024-03-25T17:31:25.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:24.999+0000 7f6999a8f640 1 -- 172.21.15.134:0/911841956 learned_addr learned my addr 172.21.15.134:0/911841956 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:25.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.004+0000 7f698affd640 10 client.?.objecter ms_handle_connect 0x7f699413f710 2024-03-25T17:31:25.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.004+0000 7f698affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:25.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.004+0000 7f698affd640 10 client.?.objecter ms_handle_connect 0x7f69940a98f0 2024-03-25T17:31:25.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.004+0000 7f698affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:25.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.010+0000 7f698affd640 10 client.?.objecter ms_handle_connect 0x7f69941430c0 2024-03-25T17:31:25.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.010+0000 7f698affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:25.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.011+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727441152 0 0) 0x7f69941d7730 con 0x7f699413f710 2024-03-25T17:31:25.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.011+0000 7f698affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:25.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.011+0000 7f698affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:25.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.011+0000 7f698affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:25.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.011+0000 7f698affd640 10 monclient(hunting): my global_id is 4315 2024-03-25T17:31:25.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.011+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6978004eb0 con 0x7f699413f710 2024-03-25T17:31:25.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.019+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116717200 0 0) 0x7f69940abdc0 con 0x7f69940a98f0 2024-03-25T17:31:25.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.019+0000 7f698affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:25.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.019+0000 7f698affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:25.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.019+0000 7f698affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:25.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.019+0000 7f698affd640 10 monclient(hunting): my global_id is 4302 2024-03-25T17:31:25.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.019+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69941d7730 con 0x7f69940a98f0 2024-03-25T17:31:25.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.031+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814036615 0 0) 0x7f6994138f10 con 0x7f69941430c0 2024-03-25T17:31:25.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.031+0000 7f698affd640 10 monclient(hunting): _negotiate 2024-03-25T17:31:25.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.031+0000 7f698affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:25.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.031+0000 7f698affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:25.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.031+0000 7f698affd640 10 monclient(hunting): my global_id is 34292 2024-03-25T17:31:25.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.031+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69940abdc0 con 0x7f69941430c0 2024-03-25T17:31:25.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.032+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2058401085 0 0) 0x7f6978004eb0 con 0x7f699413f710 2024-03-25T17:31:25.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.032+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6994138f10 con 0x7f699413f710 2024-03-25T17:31:25.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.032+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6984001bc0 con 0x7f699413f710 2024-03-25T17:31:25.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.032+0000 7f698affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:25.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.032+0000 7f698affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:25.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.032+0000 7f698affd640 10 monclient(hunting): dump: 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:25.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.032+0000 7f698affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:25.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2001005531 0 0) 0x7f6994138f10 con 0x7f699413f710 2024-03-25T17:31:25.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 >> v1:172.21.15.134:6790/0 conn(0x7f69941430c0 legacy=0x7f69941d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:25.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 >> v1:172.21.15.134:6789/0 conn(0x7f69940a98f0 legacy=0x7f6994138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:25.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 10 monclient: _finish_hunting 0 2024-03-25T17:31:25.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 1 monclient: found mon.b 2024-03-25T17:31:25.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69941390a0 con 0x7f699413f710 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 10 monclient: _finish_auth 0 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 10 monclient: _check_auth_tickets 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f698affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 5 monclient: authenticate success, global_id 4315 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 10 monclient: _renew_subs 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69941d8320 con 0x7f699413f710 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 10 monclient: _renew_subs 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:25.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69941d43c0 con 0x7f699413f710 2024-03-25T17:31:25.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.042+0000 7f699c4f4640 1 librados: init done 2024-03-25T17:31:25.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.044+0000 7f699c4f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:25.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.044+0000 7f699c4f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:25.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.044+0000 7f699c4f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:25.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.044+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69940627b0 con 0x7f699413f710 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.319+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69840011f0 con 0x7f699413f710 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.319+0000 7f698affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.319+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6984003a40 con 0x7f699413f710 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.319+0000 7f698affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.319+0000 7f698affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.319+0000 7f698affd640 10 monclient: dump: 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:25.320 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:25.321 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:25.321 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:25.321 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:25.321 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:25.321 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:25.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.319+0000 7f6998a8d640 4 set_mon_vals no callback set 2024-03-25T17:31:25.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.320+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f6984003c60 con 0x7f699413f710 2024-03-25T17:31:25.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.321+0000 7f699a290640 10 monclient: get_auth_request con 0x7f697806dda0 auth_method 0 2024-03-25T17:31:25.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.321+0000 7f698affd640 10 client.4315.objecter ms_handle_connect 0x7f697806dda0 2024-03-25T17:31:25.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.325+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f69840548d0 con 0x7f699413f710 2024-03-25T17:31:25.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.325+0000 7f698affd640 10 client.4315.objecter ms_dispatch 0x7f6994000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:25.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.325+0000 7f698affd640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:25.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.325+0000 7f698affd640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.325+0000 7f698affd640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-25T17:31:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.325+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6984050210 con 0x7f699413f710 2024-03-25T17:31:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.326+0000 7f698affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:25.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.326+0000 7f698affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:25.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.550+0000 7f699c4f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:25.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.550+0000 7f699c4f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:25.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.550+0000 7f699c4f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:25.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.550+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f699413fdc0 con 0x7f699413f710 2024-03-25T17:31:25.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.595+0000 7f698affd640 1 -- 172.21.15.134:0/911841956 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f6984052310 con 0x7f699413f710 2024-03-25T17:31:25.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.595+0000 7f698affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:25.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.595+0000 7f698affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:25.596 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:25.596 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:25.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 10 librados: watch_flush enter 2024-03-25T17:31:25.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 10 librados: watch_flush exit 2024-03-25T17:31:25.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 10 client.4315.objecter successfully canceled tick 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 >> v1:172.21.15.134:6800/610837766 conn(0x7f697806dda0 legacy=0x7f6978070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 10 monclient: shutdown 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 >> v1:172.21.15.159:6789/0 conn(0x7f699413f710 legacy=0x7f69941d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.597+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 shutdown_connections 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.598+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 >> 172.21.15.134:0/911841956 conn(0x7f69940b7ea0 msgr2=0x7f6994074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.598+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 shutdown_connections 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.598+0000 7f699c4f4640 1 -- 172.21.15.134:0/911841956 wait complete. 2024-03-25T17:31:25.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:25.598+0000 7f699c4f4640 1 librados: shutdown 2024-03-25T17:31:25.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:32.416 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.531+0000 7f37b11c1640 10 monclient: get_monmap_and_config 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.531+0000 7f37b11c1640 10 monclient: build_initial_monmap 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.531+0000 7f37b11c1640 10 monclient: monmap: 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:32.532087+0000 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:32.532087+0000 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:32.532 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:32.533 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:32.533 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:32.533 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:32.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.531+0000 7f37b11c1640 1 Processor -- start 2024-03-25T17:31:32.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.531+0000 7f37b11c1640 1 -- start start 2024-03-25T17:31:32.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.531+0000 7f37b11c1640 10 monclient: init 2024-03-25T17:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 10 monclient(hunting): picked mon.noname-b con 0x7f37ac0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 10 monclient(hunting): picked mon.noname-c con 0x7f37ac13f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 10 monclient(hunting): picked mon.noname-a con 0x7f37ac1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac142520 con 0x7f37ac1430c0 2024-03-25T17:31:32.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0b28a0 con 0x7f37ac13f710 2024-03-25T17:31:32.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0b2a30 con 0x7f37ac0a98f0 2024-03-25T17:31:32.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:32.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.532+0000 7f37b11c1640 10 monclient(hunting): authenticate will time out at 5445.795191s 2024-03-25T17:31:32.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.539+0000 7f37aad9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f37ac0a98f0 0x7f37ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50828/0 (socket says 172.21.15.134:50828) 2024-03-25T17:31:32.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.539+0000 7f37aad9d640 1 -- 172.21.15.134:0/3491830230 learned_addr learned my addr 172.21.15.134:0/3491830230 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:32.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.560+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561814886 0 0) 0x7f37ac0b2a30 con 0x7f37ac0a98f0 2024-03-25T17:31:32.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.560+0000 7f37a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:31:32.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.560+0000 7f37a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:32.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.560+0000 7f37a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:32.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.560+0000 7f37a9d9b640 10 monclient(hunting): my global_id is 34295 2024-03-25T17:31:32.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.560+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3794003520 con 0x7f37ac0a98f0 2024-03-25T17:31:32.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.567+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355246738 0 0) 0x7f37ac142520 con 0x7f37ac1430c0 2024-03-25T17:31:32.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.567+0000 7f37a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:31:32.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.567+0000 7f37a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:32.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.567+0000 7f37a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:32.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.567+0000 7f37a9d9b640 10 monclient(hunting): my global_id is 4305 2024-03-25T17:31:32.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.567+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac0b2a30 con 0x7f37ac1430c0 2024-03-25T17:31:32.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.571+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836489604 0 0) 0x7f37ac0b28a0 con 0x7f37ac13f710 2024-03-25T17:31:32.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.571+0000 7f37a9d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:31:32.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.571+0000 7f37a9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:32.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.571+0000 7f37a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:32.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.571+0000 7f37a9d9b640 10 monclient(hunting): my global_id is 4318 2024-03-25T17:31:32.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.571+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac142520 con 0x7f37ac13f710 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927762037 0 0) 0x7f3794003520 con 0x7f37ac0a98f0 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 >> v1:172.21.15.159:6789/0 conn(0x7f37ac13f710 legacy=0x7f37ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 >> v1:172.21.15.134:6789/0 conn(0x7f37ac1430c0 legacy=0x7f37ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 1 monclient: found mon.noname-b 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:31:32.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac0b44c0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 10 monclient: _finish_auth 0 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f379c001690 con 0x7f37ac0a98f0 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37a9d9b640 10 monclient: dump: 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:32.587 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:32.588 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:32.588 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:32.588 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:32.588 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:32.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37b11c1640 5 monclient: authenticate success, global_id 34295 2024-03-25T17:31:32.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.585+0000 7f37b11c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f379c0017f0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37a9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37a9d9b640 1 -- 172.21.15.134:0/3491830230 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f379c001c00 con 0x7f37ac0a98f0 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37b11c1640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37b11c1640 4 set_mon_vals no callback set 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37a9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37a9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.603+0000 7f37a9d9b640 10 monclient: dump: 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.604 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:32.605 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:32.605 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:32.605 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:32.605 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:32.605 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.604+0000 7f37b11c1640 10 monclient: shutdown 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.604+0000 7f37b11c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.604+0000 7f37b11c1640 1 -- 172.21.15.134:0/3491830230 >> v1:172.21.15.134:6790/0 conn(0x7f37ac0a98f0 legacy=0x7f37ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.604+0000 7f37b11c1640 1 -- 172.21.15.134:0/3491830230 shutdown_connections 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.604+0000 7f37b11c1640 1 -- 172.21.15.134:0/3491830230 >> 172.21.15.134:0/3491830230 conn(0x7f37ac0b7ea0 msgr2=0x7f37ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.604+0000 7f37b11c1640 1 -- 172.21.15.134:0/3491830230 shutdown_connections 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.604+0000 7f37b11c1640 1 -- 172.21.15.134:0/3491830230 wait complete. 2024-03-25T17:31:32.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 10 monclient: build_initial_monmap 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 10 monclient: monmap: 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 1 librados: starting msgr at 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 1 librados: starting objecter 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 1 Processor -- start 2024-03-25T17:31:32.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 1 -- start start 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 1 librados: setting wanted keys 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 1 librados: calling monclient init 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.605+0000 7f37b11c1640 10 monclient: init 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 10 monclient(hunting): picked mon.noname-b con 0x7f37ac0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 10 monclient(hunting): picked mon.noname-a con 0x7f37ac13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 10 monclient(hunting): picked mon.noname-c con 0x7f37ac1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac138b20 con 0x7f37ac13f710 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac137a80 con 0x7f37ac0a98f0 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac139880 con 0x7f37ac1430c0 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:32.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.606+0000 7f37b11c1640 10 monclient(hunting): authenticate will time out at 5445.869179s 2024-03-25T17:31:32.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.619+0000 7f37aad9d640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f37ac0a98f0 0x7f37ac137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:48632/0 (socket says 172.21.15.134:48632) 2024-03-25T17:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.619+0000 7f37aad9d640 1 -- 172.21.15.134:0/2076800715 learned_addr learned my addr 172.21.15.134:0/2076800715 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.620+0000 7f37937fe640 10 client.?.objecter ms_handle_connect 0x7f37ac0a98f0 2024-03-25T17:31:32.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.620+0000 7f37937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:32.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.622+0000 7f37937fe640 10 client.?.objecter ms_handle_connect 0x7f37ac13f710 2024-03-25T17:31:32.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.622+0000 7f37937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:32.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.626+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371669546 0 0) 0x7f37ac137a80 con 0x7f37ac0a98f0 2024-03-25T17:31:32.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.626+0000 7f37937fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:32.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.626+0000 7f37937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:32.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.626+0000 7f37937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:32.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.626+0000 7f37937fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T17:31:32.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.626+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798004eb0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.638+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424916425 0 0) 0x7f37ac138b20 con 0x7f37ac13f710 2024-03-25T17:31:32.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.638+0000 7f37937fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:32.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.638+0000 7f37937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:32.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.638+0000 7f37937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:32.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.638+0000 7f37937fe640 10 monclient(hunting): my global_id is 4308 2024-03-25T17:31:32.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.638+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac137a80 con 0x7f37ac13f710 2024-03-25T17:31:32.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.639+0000 7f37937fe640 10 client.?.objecter ms_handle_connect 0x7f37ac1430c0 2024-03-25T17:31:32.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.639+0000 7f37937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.643+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4259838183 0 0) 0x7f3798004eb0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.643+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ac138b20 con 0x7f37ac0a98f0 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.643+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f379c001bc0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.643+0000 7f37937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.643+0000 7f37937fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.643+0000 7f37937fe640 10 monclient(hunting): dump: 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:32.644 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:32.645 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:32.645 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:32.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.643+0000 7f37937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:32.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.648+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3118733106 0 0) 0x7f37ac139880 con 0x7f37ac1430c0 2024-03-25T17:31:32.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.648+0000 7f37937fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:32.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.648+0000 7f37937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:32.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.648+0000 7f37937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:32.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.648+0000 7f37937fe640 10 monclient(hunting): my global_id is 34298 2024-03-25T17:31:32.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.648+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798004eb0 con 0x7f37ac1430c0 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3570238138 0 0) 0x7f37ac138b20 con 0x7f37ac0a98f0 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 >> v1:172.21.15.134:6790/0 conn(0x7f37ac1430c0 legacy=0x7f37ac1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 >> v1:172.21.15.134:6789/0 conn(0x7f37ac13f710 legacy=0x7f37ac139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 10 monclient: _finish_hunting 0 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 1 monclient: found mon.b 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac139a10 con 0x7f37ac0a98f0 2024-03-25T17:31:32.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 10 monclient: _finish_auth 0 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 10 monclient: _check_auth_tickets 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 5 monclient: authenticate success, global_id 4321 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 10 monclient: _renew_subs 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac1dab00 con 0x7f37ac0a98f0 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 10 monclient: _renew_subs 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac13b4f0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.657+0000 7f37b11c1640 1 librados: init done 2024-03-25T17:31:32.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.658+0000 7f37b11c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:32.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.658+0000 7f37b11c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:32.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.658+0000 7f37b11c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:32.660 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.658+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac0627b0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.957+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f379c0011f0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.957+0000 7f37937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.957+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f379c004cb0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.957+0000 7f37937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.957+0000 7f37937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.957+0000 7f37937fe640 10 monclient: dump: 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:32.958 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:32.959 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:32.959 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:32.959 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:32.959 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.957+0000 7f37a959a640 4 set_mon_vals no callback set 2024-03-25T17:31:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.958+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f379c01ad00 con 0x7f37ac0a98f0 2024-03-25T17:31:32.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.959+0000 7f37aa59c640 10 monclient: get_auth_request con 0x7f379806ddb0 auth_method 0 2024-03-25T17:31:32.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.959+0000 7f37937fe640 10 client.4321.objecter ms_handle_connect 0x7f379806ddb0 2024-03-25T17:31:32.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.964+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f379c054970 con 0x7f37ac0a98f0 2024-03-25T17:31:32.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.964+0000 7f37937fe640 10 client.4321.objecter ms_dispatch 0x7f37ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:32.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.964+0000 7f37937fe640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:32.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.964+0000 7f37937fe640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:32.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.964+0000 7f37937fe640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T17:31:32.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.965+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f379c0502b0 con 0x7f37ac0a98f0 2024-03-25T17:31:32.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.965+0000 7f37937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:32.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:32.965+0000 7f37937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:33.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.183+0000 7f37b11c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:33.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.183+0000 7f37b11c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:33.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.183+0000 7f37b11c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:33.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.183+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37ac0b3e80 con 0x7f37ac0a98f0 2024-03-25T17:31:33.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.206+0000 7f37937fe640 1 -- 172.21.15.134:0/2076800715 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f379c0523b0 con 0x7f37ac0a98f0 2024-03-25T17:31:33.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.206+0000 7f37937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:33.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.206+0000 7f37937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:33.207 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:33.207 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:33.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 10 librados: watch_flush enter 2024-03-25T17:31:33.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 10 librados: watch_flush exit 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 10 client.4321.objecter successfully canceled tick 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 >> v1:172.21.15.134:6800/610837766 conn(0x7f379806ddb0 legacy=0x7f3798070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 10 monclient: shutdown 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 >> v1:172.21.15.159:6789/0 conn(0x7f37ac0a98f0 legacy=0x7f37ac137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 shutdown_connections 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 >> 172.21.15.134:0/2076800715 conn(0x7f37ac0b7ea0 msgr2=0x7f37ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 shutdown_connections 2024-03-25T17:31:33.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 1 -- 172.21.15.134:0/2076800715 wait complete. 2024-03-25T17:31:33.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:33.209+0000 7f37b11c1640 1 librados: shutdown 2024-03-25T17:31:33.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:40.029 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:40.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.146+0000 7f740ae16640 10 monclient: get_monmap_and_config 2024-03-25T17:31:40.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.146+0000 7f740ae16640 10 monclient: build_initial_monmap 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.146+0000 7f740ae16640 10 monclient: monmap: 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:40.147211+0000 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:40.147211+0000 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:40.147 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 1 Processor -- start 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 1 -- start start 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient: init 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient(hunting): picked mon.noname-a con 0x7f74040a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient(hunting): picked mon.noname-b con 0x7f740413f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient(hunting): picked mon.noname-c con 0x7f74041430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7404142520 con 0x7f74040a98f0 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74040b28a0 con 0x7f74041430c0 2024-03-25T17:31:40.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74040b2a30 con 0x7f740413f710 2024-03-25T17:31:40.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:40.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.147+0000 7f740ae16640 10 monclient(hunting): authenticate will time out at 5453.410516s 2024-03-25T17:31:40.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.150+0000 7f7408bb2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f74040a98f0 0x7f74040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:59108/0 (socket says 172.21.15.134:59108) 2024-03-25T17:31:40.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.151+0000 7f7408bb2640 1 -- 172.21.15.134:0/861692086 learned_addr learned my addr 172.21.15.134:0/861692086 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:40.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.156+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1129971754 0 0) 0x7f7404142520 con 0x7f74040a98f0 2024-03-25T17:31:40.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.156+0000 7f73fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:40.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.156+0000 7f73fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:40.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.156+0000 7f73fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:40.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.156+0000 7f73fb7fe640 10 monclient(hunting): my global_id is 4311 2024-03-25T17:31:40.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.156+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ec003520 con 0x7f74040a98f0 2024-03-25T17:31:40.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470864879 0 0) 0x7f73ec003520 con 0x7f74040a98f0 2024-03-25T17:31:40.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 >> v1:172.21.15.134:6790/0 conn(0x7f740413f710 legacy=0x7f74041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:31:40.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 >> v1:172.21.15.159:6789/0 conn(0x7f74041430c0 legacy=0x7f740414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:40.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:31:40.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 1 monclient: found mon.noname-a 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74040b44c0 con 0x7f74040a98f0 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 10 monclient: _finish_auth 0 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7400001690 con 0x7f74040a98f0 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f73fb7fe640 10 monclient: dump: 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:40.175 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f740ae16640 5 monclient: authenticate success, global_id 4311 2024-03-25T17:31:40.176 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.173+0000 7f740ae16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7400001860 con 0x7f74040a98f0 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f73fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f73fb7fe640 1 -- 172.21.15.134:0/861692086 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7400001c70 con 0x7f74040a98f0 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f740ae16640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f740ae16640 4 set_mon_vals no callback set 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f73fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f73fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.192+0000 7f73fb7fe640 10 monclient: dump: 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:40.193 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:40.194 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.193+0000 7f740ae16640 10 monclient: shutdown 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.193+0000 7f740ae16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.193+0000 7f740ae16640 1 -- 172.21.15.134:0/861692086 >> v1:172.21.15.134:6789/0 conn(0x7f74040a98f0 legacy=0x7f74040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.193+0000 7f740ae16640 1 -- 172.21.15.134:0/861692086 shutdown_connections 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.193+0000 7f740ae16640 1 -- 172.21.15.134:0/861692086 >> 172.21.15.134:0/861692086 conn(0x7f74040b7ea0 msgr2=0x7f74040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.193+0000 7f740ae16640 1 -- 172.21.15.134:0/861692086 shutdown_connections 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.193+0000 7f740ae16640 1 -- 172.21.15.134:0/861692086 wait complete. 2024-03-25T17:31:40.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 10 monclient: build_initial_monmap 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 10 monclient: monmap: 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 1 librados: starting msgr at 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 1 librados: starting objecter 2024-03-25T17:31:40.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 1 Processor -- start 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 1 -- start start 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 1 librados: setting wanted keys 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 1 librados: calling monclient init 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.194+0000 7f740ae16640 10 monclient: init 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 10 monclient(hunting): picked mon.noname-b con 0x7f74040a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 10 monclient(hunting): picked mon.noname-a con 0x7f740413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 10 monclient(hunting): picked mon.noname-c con 0x7f74041430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74040b7320 con 0x7f740413f710 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740413b3f0 con 0x7f74040a98f0 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740413b6d0 con 0x7f74041430c0 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:40.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.195+0000 7f740ae16640 10 monclient(hunting): authenticate will time out at 5453.458480s 2024-03-25T17:31:40.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.213+0000 7f7408bb2640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f74040a98f0 0x7f7404139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:41064/0 (socket says 172.21.15.134:41064) 2024-03-25T17:31:40.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.213+0000 7f7408bb2640 1 -- 172.21.15.134:0/1018964642 learned_addr learned my addr 172.21.15.134:0/1018964642 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.215+0000 7f73f97fa640 10 client.?.objecter ms_handle_connect 0x7f74040a98f0 2024-03-25T17:31:40.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.215+0000 7f73f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:40.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.220+0000 7f73f97fa640 10 client.?.objecter ms_handle_connect 0x7f74041430c0 2024-03-25T17:31:40.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.220+0000 7f73f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:40.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.225+0000 7f73f97fa640 10 client.?.objecter ms_handle_connect 0x7f740413f710 2024-03-25T17:31:40.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.225+0000 7f73f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:40.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239160485 0 0) 0x7f740413b3f0 con 0x7f74040a98f0 2024-03-25T17:31:40.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:40.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:40.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): my global_id is 4327 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f0004eb0 con 0x7f74040a98f0 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982049259 0 0) 0x7f740413b6d0 con 0x7f74041430c0 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 10 monclient(hunting): my global_id is 34301 2024-03-25T17:31:40.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.231+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740413b3f0 con 0x7f74041430c0 2024-03-25T17:31:40.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.241+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093973439 0 0) 0x7f74040b7320 con 0x7f740413f710 2024-03-25T17:31:40.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.241+0000 7f73f97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:31:40.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.241+0000 7f73f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:40.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.241+0000 7f73f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:40.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.241+0000 7f73f97fa640 10 monclient(hunting): my global_id is 4314 2024-03-25T17:31:40.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.241+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740413b6d0 con 0x7f740413f710 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.249+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3932969536 0 0) 0x7f73f0004eb0 con 0x7f74040a98f0 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.249+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74040b7320 con 0x7f74040a98f0 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.249+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7400001b70 con 0x7f74040a98f0 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.249+0000 7f73f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.249+0000 7f73f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.249+0000 7f73f97fa640 10 monclient(hunting): dump: 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.250 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:40.251 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:40.251 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:40.251 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:40.251 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:40.251 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.249+0000 7f73f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:40.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.257+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2204439680 0 0) 0x7f740413b6d0 con 0x7f740413f710 2024-03-25T17:31:40.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.257+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f0004eb0 con 0x7f740413f710 2024-03-25T17:31:40.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.257+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f73f40017d0 con 0x7f740413f710 2024-03-25T17:31:40.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.257+0000 7f73f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.257+0000 7f73f97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.257+0000 7f73f97fa640 10 monclient(hunting): dump: 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:40.258 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2945011352 0 0) 0x7f740413b3f0 con 0x7f74041430c0 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f740413b6d0 con 0x7f74041430c0 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f73fc001b10 con 0x7f74041430c0 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 10 monclient(hunting): dump: 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:40.265 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1748829053 0 0) 0x7f73f0004eb0 con 0x7f740413f710 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 >> v1:172.21.15.134:6790/0 conn(0x7f74041430c0 legacy=0x7f74041e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 >> v1:172.21.15.159:6789/0 conn(0x7f74040a98f0 legacy=0x7f7404139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 10 monclient: _finish_hunting 0 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 monclient: found mon.a 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74041302d0 con 0x7f740413f710 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 10 monclient: _finish_auth 0 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 10 monclient: _check_auth_tickets 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f73f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 5 monclient: authenticate success, global_id 4314 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 10 monclient: _renew_subs 2024-03-25T17:31:40.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7404139880 con 0x7f740413f710 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 10 monclient: _renew_subs 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f740413bcd0 con 0x7f740413f710 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.263+0000 7f740ae16640 1 librados: init done 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.264+0000 7f740ae16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.264+0000 7f740ae16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.264+0000 7f740ae16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:40.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.264+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740414f2d0 con 0x7f740413f710 2024-03-25T17:31:40.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.577+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73f4001cc0 con 0x7f740413f710 2024-03-25T17:31:40.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.577+0000 7f73f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:40.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.577+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f73f4003790 con 0x7f740413f710 2024-03-25T17:31:40.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.577+0000 7f73f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:40.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.577+0000 7f73f97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:31:40.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.577+0000 7f73f97fa640 10 monclient: dump: 2024-03-25T17:31:40.578 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:40.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.577+0000 7f73faffd640 4 set_mon_vals no callback set 2024-03-25T17:31:40.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.578+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f73f4019760 con 0x7f740413f710 2024-03-25T17:31:40.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.579+0000 7f7408bb2640 10 monclient: get_auth_request con 0x7f73f006e080 auth_method 0 2024-03-25T17:31:40.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.579+0000 7f73f97fa640 10 client.4314.objecter ms_handle_connect 0x7f73f006e080 2024-03-25T17:31:40.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f73f4053400 con 0x7f740413f710 2024-03-25T17:31:40.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 10 client.4314.objecter ms_dispatch 0x7f7404000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:40.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:40.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:40.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-25T17:31:40.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73f404ef10 con 0x7f740413f710 2024-03-25T17:31:40.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:40.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.584+0000 7f73f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:40.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.822+0000 7f740ae16640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:40.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.822+0000 7f740ae16640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:40.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.822+0000 7f740ae16640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:31:40.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.822+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74040627b0 con 0x7f740413f710 2024-03-25T17:31:40.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.846+0000 7f73f97fa640 1 -- 172.21.15.134:0/1018964642 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f73f4050e40 con 0x7f740413f710 2024-03-25T17:31:40.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.846+0000 7f73f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:40.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.846+0000 7f73f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:40.847 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:40.847 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:40.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 10 librados: watch_flush enter 2024-03-25T17:31:40.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 10 librados: watch_flush exit 2024-03-25T17:31:40.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 10 client.4314.objecter successfully canceled tick 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 >> v1:172.21.15.134:6800/610837766 conn(0x7f73f006e080 legacy=0x7f73f0070540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 10 monclient: shutdown 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 >> v1:172.21.15.134:6789/0 conn(0x7f740413f710 legacy=0x7f74041df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 shutdown_connections 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.848+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 >> 172.21.15.134:0/1018964642 conn(0x7f74040b7ea0 msgr2=0x7f7404074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.849+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 shutdown_connections 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.849+0000 7f740ae16640 1 -- 172.21.15.134:0/1018964642 wait complete. 2024-03-25T17:31:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:40.849+0000 7f740ae16640 1 librados: shutdown 2024-03-25T17:31:40.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:44.319 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:49.335 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.376 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:49.393 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.406 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:02:49.423 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.427 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:49.443 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.437 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:49.453 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.445 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:49.474 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.452 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:49.480 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.463 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:02:49.480 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:44.477 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:02:49.506 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:31:47.666 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:47.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.782+0000 7f93810d6640 10 monclient: get_monmap_and_config 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.782+0000 7f93810d6640 10 monclient: build_initial_monmap 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.782+0000 7f93810d6640 10 monclient: monmap: 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:47.783458+0000 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:47.783458+0000 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:47.783 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:47.784 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 1 Processor -- start 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 1 -- start start 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient: init 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient(hunting): picked mon.noname-b con 0x7f937c0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient(hunting): picked mon.noname-c con 0x7f937c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient(hunting): picked mon.noname-a con 0x7f937c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0af940 con 0x7f937c14be90 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0b0a70 con 0x7f937c0b1730 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0b0d50 con 0x7f937c0b4d60 2024-03-25T17:31:47.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:47.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.783+0000 7f93810d6640 10 monclient(hunting): authenticate will time out at 5461.046637s 2024-03-25T17:31:47.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.788+0000 7f937ad9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f937c0b4d60 0x7f937c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:57810/0 (socket says 172.21.15.134:57810) 2024-03-25T17:31:47.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.788+0000 7f937ad9d640 1 -- 172.21.15.134:0/927490779 learned_addr learned my addr 172.21.15.134:0/927490779 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:47.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.792+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521684090 0 0) 0x7f937c0b0d50 con 0x7f937c0b4d60 2024-03-25T17:31:47.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.792+0000 7f9379d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:31:47.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.792+0000 7f9379d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:47.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.792+0000 7f9379d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:47.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.792+0000 7f9379d9b640 10 monclient(hunting): my global_id is 34304 2024-03-25T17:31:47.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.792+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9364003520 con 0x7f937c0b4d60 2024-03-25T17:31:47.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.794+0000 7f937a59c640 1 -- 172.21.15.134:0/927490779 >> v1:172.21.15.159:6789/0 conn(0x7f937c0b1730 legacy=0x7f937c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T17:31:47.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.794+0000 7f937a59c640 1 -- 172.21.15.134:0/927490779 >> v1:172.21.15.159:6789/0 conn(0x7f937c0b1730 legacy=0x7f937c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T17:31:47.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.794+0000 7f937a59c640 1 --1- 172.21.15.134:0/927490779 >> v1:172.21.15.159:6789/0 conn(0x7f937c0b1730 0x7f937c0b1be0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T17:31:47.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597491971 0 0) 0x7f9364003520 con 0x7f937c0b4d60 2024-03-25T17:31:47.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 >> v1:172.21.15.159:6789/0 conn(0x7f937c0b1730 legacy=0x7f937c0b1be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T17:31:47.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 >> v1:172.21.15.134:6789/0 conn(0x7f937c14be90 legacy=0x7f937c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:31:47.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:31:47.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 1 monclient: found mon.noname-b 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c0b1030 con 0x7f937c0b4d60 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 10 monclient: _finish_auth 0 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f936c001690 con 0x7f937c0b4d60 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f9379d9b640 10 monclient: dump: 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:47.801 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f93810d6640 5 monclient: authenticate success, global_id 34304 2024-03-25T17:31:47.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.800+0000 7f93810d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:47.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f936c001860 con 0x7f937c0b4d60 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f9379d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f9379d9b640 1 -- 172.21.15.134:0/927490779 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f936c001cb0 con 0x7f937c0b4d60 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f93810d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f93810d6640 4 set_mon_vals no callback set 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f9379d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f9379d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.829+0000 7f9379d9b640 10 monclient: dump: 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.830 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.831 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:47.831 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:47.831 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:47.831 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:47.831 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:47.831 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:47.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.830+0000 7f93810d6640 10 monclient: shutdown 2024-03-25T17:31:47.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.830+0000 7f93810d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:47.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.830+0000 7f93810d6640 1 -- 172.21.15.134:0/927490779 >> v1:172.21.15.134:6790/0 conn(0x7f937c0b4d60 legacy=0x7f937c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:47.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.830+0000 7f93810d6640 1 -- 172.21.15.134:0/927490779 shutdown_connections 2024-03-25T17:31:47.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.830+0000 7f93810d6640 1 -- 172.21.15.134:0/927490779 >> 172.21.15.134:0/927490779 conn(0x7f937c0617a0 msgr2=0x7f937c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:47.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.830+0000 7f93810d6640 1 -- 172.21.15.134:0/927490779 shutdown_connections 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.830+0000 7f93810d6640 1 -- 172.21.15.134:0/927490779 wait complete. 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 10 monclient: build_initial_monmap 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 10 monclient: monmap: 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:47.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 1 librados: starting msgr at 2024-03-25T17:31:47.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 1 librados: starting objecter 2024-03-25T17:31:47.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 1 Processor -- start 2024-03-25T17:31:47.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 1 -- start start 2024-03-25T17:31:47.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 1 librados: setting wanted keys 2024-03-25T17:31:47.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 1 librados: calling monclient init 2024-03-25T17:31:47.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.831+0000 7f93810d6640 10 monclient: init 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 10 monclient(hunting): picked mon.noname-a con 0x7f937c0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 10 monclient(hunting): picked mon.noname-b con 0x7f937c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 10 monclient(hunting): picked mon.noname-c con 0x7f937c14be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c13ba90 con 0x7f937c0b1730 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0a9470 con 0x7f937c0b4d60 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0a9600 con 0x7f937c14be90 2024-03-25T17:31:47.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:47.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.832+0000 7f93810d6640 10 monclient(hunting): authenticate will time out at 5461.095537s 2024-03-25T17:31:47.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.844+0000 7f937a59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f937c0b4d60 0x7f937c1da940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:45438/0 (socket says 172.21.15.134:45438) 2024-03-25T17:31:47.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.844+0000 7f937a59c640 1 -- 172.21.15.134:0/2359589454 learned_addr learned my addr 172.21.15.134:0/2359589454 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:47.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.844+0000 7f93637fe640 10 client.?.objecter ms_handle_connect 0x7f937c0b4d60 2024-03-25T17:31:47.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.844+0000 7f93637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:47.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.846+0000 7f93637fe640 10 client.?.objecter ms_handle_connect 0x7f937c0b1730 2024-03-25T17:31:47.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.846+0000 7f93637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:47.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.851+0000 7f93637fe640 10 client.?.objecter ms_handle_connect 0x7f937c14be90 2024-03-25T17:31:47.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.851+0000 7f93637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.853+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346442227 0 0) 0x7f937c13ba90 con 0x7f937c0b1730 2024-03-25T17:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.853+0000 7f93637fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.853+0000 7f93637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.853+0000 7f93637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.853+0000 7f93637fe640 10 monclient(hunting): my global_id is 4317 2024-03-25T17:31:47.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.853+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9368004eb0 con 0x7f937c0b1730 2024-03-25T17:31:47.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.857+0000 7f937b59e640 1 -- 172.21.15.134:0/2359589454 >> v1:172.21.15.134:6790/0 conn(0x7f937c14be90 legacy=0x7f937c1db050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T17:31:47.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.857+0000 7f937b59e640 1 -- 172.21.15.134:0/2359589454 >> v1:172.21.15.134:6790/0 conn(0x7f937c14be90 legacy=0x7f937c1db050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:31:47.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.857+0000 7f937b59e640 1 --1- 172.21.15.134:0/2359589454 >> v1:172.21.15.134:6790/0 conn(0x7f937c14be90 0x7f937c1db050 :-1 s=OPENED pgs=13515 cs=1 l=1).handle_message read tag failed 2024-03-25T17:31:47.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.857+0000 7f937b59e640 1 --1- 172.21.15.134:0/2359589454 >> v1:172.21.15.134:6790/0 conn(0x7f937c14be90 0x7f937c1db050 :-1 s=OPENED pgs=13515 cs=1 l=1).fault on lossy channel, failing 2024-03-25T17:31:47.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.857+0000 7f93637fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.134:6790/0 2024-03-25T17:31:47.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.863+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282559213 0 0) 0x7f937c0a9470 con 0x7f937c0b4d60 2024-03-25T17:31:47.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.863+0000 7f93637fe640 10 monclient(hunting): _negotiate 2024-03-25T17:31:47.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.863+0000 7f93637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:47.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.863+0000 7f93637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:47.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.863+0000 7f93637fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T17:31:47.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.863+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f937c13ba90 con 0x7f937c0b4d60 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.874+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2007185477 0 0) 0x7f9368004eb0 con 0x7f937c0b1730 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.874+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f937c0a9470 con 0x7f937c0b1730 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.874+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f936c001b70 con 0x7f937c0b1730 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.874+0000 7f93637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.874+0000 7f93637fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.874+0000 7f93637fe640 10 monclient(hunting): dump: 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.875 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.876 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:47.876 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:47.876 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:47.876 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:47.876 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:47.876 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:47.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.874+0000 7f93637fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:47.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.886+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4146903280 0 0) 0x7f937c13ba90 con 0x7f937c0b4d60 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.886+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9368004eb0 con 0x7f937c0b4d60 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.886+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9370004000 con 0x7f937c0b4d60 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.886+0000 7f93637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.886+0000 7f93637fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.886+0000 7f93637fe640 10 monclient(hunting): dump: 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:47.887 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:47.888 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2704999078 0 0) 0x7f9368004eb0 con 0x7f937c0b4d60 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 >> v1:172.21.15.134:6790/0 conn(0x7f937c14be90 legacy=0x7f937c1db050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 >> v1:172.21.15.134:6789/0 conn(0x7f937c0b1730 legacy=0x7f937c137e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 10 monclient: _finish_hunting 0 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 1 monclient: found mon.b 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c138510 con 0x7f937c0b4d60 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 10 monclient: _finish_auth 0 2024-03-25T17:31:47.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 10 monclient: _check_auth_tickets 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 5 monclient: authenticate success, global_id 4330 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 10 monclient: _renew_subs 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c138760 con 0x7f937c0b4d60 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 10 monclient: _renew_subs 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c143080 con 0x7f937c0b4d60 2024-03-25T17:31:47.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.901+0000 7f93810d6640 1 librados: init done 2024-03-25T17:31:47.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.902+0000 7f93810d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:47.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.902+0000 7f93810d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:47.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.902+0000 7f93810d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:47.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.902+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c0aa5a0 con 0x7f937c0b4d60 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.946+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9370000e80 con 0x7f937c0b4d60 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.946+0000 7f93637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.946+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9370001070 con 0x7f937c0b4d60 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.946+0000 7f93637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.946+0000 7f93637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.946+0000 7f93637fe640 10 monclient: dump: 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:47.947 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:47.948 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:47.948 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:47.948 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:47.948 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:47.948 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:47.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.946+0000 7f937959a640 4 set_mon_vals no callback set 2024-03-25T17:31:47.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.947+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f9370001290 con 0x7f937c0b4d60 2024-03-25T17:31:47.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.948+0000 7f937ad9d640 10 monclient: get_auth_request con 0x7f936806de10 auth_method 0 2024-03-25T17:31:47.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:47.948+0000 7f93637fe640 10 client.4330.objecter ms_handle_connect 0x7f936806de10 2024-03-25T17:31:48.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.205+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f9370054550 con 0x7f937c0b4d60 2024-03-25T17:31:48.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.206+0000 7f93637fe640 10 client.4330.objecter ms_dispatch 0x7f937c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:48.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.206+0000 7f93637fe640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:48.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.206+0000 7f93637fe640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:48.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.206+0000 7f93637fe640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T17:31:48.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.206+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9370050060 con 0x7f937c0b4d60 2024-03-25T17:31:48.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.206+0000 7f93637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:48.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.206+0000 7f93637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:48.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.450+0000 7f93810d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:48.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.451+0000 7f93810d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:48.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.451+0000 7f93810d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:48.452 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.451+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f937c0b22f0 con 0x7f937c0b4d60 2024-03-25T17:31:48.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.532+0000 7f93637fe640 1 -- 172.21.15.134:0/2359589454 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f937c0b22f0 con 0x7f937c0b4d60 2024-03-25T17:31:48.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.532+0000 7f93637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:48.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.532+0000 7f93637fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:48.533 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:48.533 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 10 librados: watch_flush enter 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 10 librados: watch_flush exit 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 10 client.4330.objecter successfully canceled tick 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 >> v1:172.21.15.134:6800/610837766 conn(0x7f936806de10 legacy=0x7f93680702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 10 monclient: shutdown 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.534+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 >> v1:172.21.15.159:6789/0 conn(0x7f937c0b4d60 legacy=0x7f937c1da940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.535+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 shutdown_connections 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.535+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 >> 172.21.15.134:0/2359589454 conn(0x7f937c0617a0 msgr2=0x7f937c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.535+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 shutdown_connections 2024-03-25T17:31:48.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.535+0000 7f93810d6640 1 -- 172.21.15.134:0/2359589454 wait complete. 2024-03-25T17:31:48.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:48.535+0000 7f93810d6640 1 librados: shutdown 2024-03-25T17:31:48.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:31:55.357 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:31:55.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.477+0000 7f578f8e6640 10 monclient: get_monmap_and_config 2024-03-25T17:31:55.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.477+0000 7f578f8e6640 10 monclient: build_initial_monmap 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.477+0000 7f578f8e6640 10 monclient: monmap: 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:31:55.478875+0000 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:31:55.478875+0000 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:31:55.479 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:55.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.478+0000 7f578f8e6640 1 Processor -- start 2024-03-25T17:31:55.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 1 -- start start 2024-03-25T17:31:55.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient: init 2024-03-25T17:31:55.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:55.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:31:55.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f57880a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:55.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f578813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:55.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f57881430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:55.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5788142520 con 0x7f578813f710 2024-03-25T17:31:55.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880b28a0 con 0x7f57880a98f0 2024-03-25T17:31:55.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880b2a30 con 0x7f57881430c0 2024-03-25T17:31:55.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:55.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.479+0000 7f578f8e6640 10 monclient(hunting): authenticate will time out at 5468.742750s 2024-03-25T17:31:55.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.489+0000 7f578de83640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f57881430c0 0x7f578814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:37600/0 (socket says 172.21.15.134:37600) 2024-03-25T17:31:55.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.489+0000 7f578de83640 1 -- 172.21.15.134:0/3057956395 learned_addr learned my addr 172.21.15.134:0/3057956395 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:55.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.505+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327731658 0 0) 0x7f57880b2a30 con 0x7f57881430c0 2024-03-25T17:31:55.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.505+0000 7f577ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:31:55.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.505+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:55.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.505+0000 7f577ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:55.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.505+0000 7f577ffff640 10 monclient(hunting): my global_id is 34307 2024-03-25T17:31:55.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.505+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5774003520 con 0x7f57881430c0 2024-03-25T17:31:55.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.516+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771126582 0 0) 0x7f57880b28a0 con 0x7f57880a98f0 2024-03-25T17:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.516+0000 7f577ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.516+0000 7f577ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.516+0000 7f577ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.516+0000 7f577ffff640 10 monclient(hunting): my global_id is 4333 2024-03-25T17:31:55.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.516+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57880b2a30 con 0x7f57880a98f0 2024-03-25T17:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069713597 0 0) 0x7f57880b2a30 con 0x7f57880a98f0 2024-03-25T17:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 >> v1:172.21.15.134:6790/0 conn(0x7f57881430c0 legacy=0x7f578814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 >> v1:172.21.15.134:6789/0 conn(0x7f578813f710 legacy=0x7f57881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:55.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 1 monclient: found mon.noname-c 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57880b44c0 con 0x7f57880a98f0 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 10 monclient: _finish_auth 0 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5784001670 con 0x7f57880a98f0 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f577ffff640 10 monclient: dump: 2024-03-25T17:31:55.528 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f578f8e6640 5 monclient: authenticate success, global_id 4333 2024-03-25T17:31:55.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.526+0000 7f578f8e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57840017d0 con 0x7f57880a98f0 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f577ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f577ffff640 1 -- 172.21.15.134:0/3057956395 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5784001be0 con 0x7f57880a98f0 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f577ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f577ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f577ffff640 10 monclient: dump: 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:55.538 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f578f8e6640 10 monclient: get_monmap_and_config success 2024-03-25T17:31:55.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.537+0000 7f578f8e6640 4 set_mon_vals no callback set 2024-03-25T17:31:55.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.538+0000 7f578f8e6640 10 monclient: shutdown 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.538+0000 7f578f8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.538+0000 7f578f8e6640 1 -- 172.21.15.134:0/3057956395 >> v1:172.21.15.159:6789/0 conn(0x7f57880a98f0 legacy=0x7f57880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.538+0000 7f578f8e6640 1 -- 172.21.15.134:0/3057956395 shutdown_connections 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.538+0000 7f578f8e6640 1 -- 172.21.15.134:0/3057956395 >> 172.21.15.134:0/3057956395 conn(0x7f57880b7ea0 msgr2=0x7f57880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.538+0000 7f578f8e6640 1 -- 172.21.15.134:0/3057956395 shutdown_connections 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.538+0000 7f578f8e6640 1 -- 172.21.15.134:0/3057956395 wait complete. 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.539+0000 7f578f8e6640 10 monclient: build_initial_monmap 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.539+0000 7f578f8e6640 10 monclient: monmap: 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:31:55.542 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.539+0000 7f578f8e6640 1 librados: starting msgr at 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.539+0000 7f578f8e6640 1 librados: starting objecter 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.539+0000 7f578f8e6640 1 Processor -- start 2024-03-25T17:31:55.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 1 -- start start 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 1 librados: setting wanted keys 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 1 librados: calling monclient init 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 10 monclient: init 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 10 monclient: _reopen_session rank -1 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f57880a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f578813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f57881430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880ab9c0 con 0x7f57881430c0 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.540+0000 7f578f8e6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57880abc30 con 0x7f57880a98f0 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.541+0000 7f578f8e6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57881daba0 con 0x7f578813f710 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.541+0000 7f578f8e6640 10 monclient(hunting): _renew_subs 2024-03-25T17:31:55.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.541+0000 7f578f8e6640 10 monclient(hunting): authenticate will time out at 5468.803838s 2024-03-25T17:31:55.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.555+0000 7f578ce81640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f578813f710 0x7f57881db0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:37656/0 (socket says 172.21.15.134:37656) 2024-03-25T17:31:55.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.555+0000 7f578ce81640 1 -- 172.21.15.134:0/1221285464 learned_addr learned my addr 172.21.15.134:0/1221285464 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:31:55.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.556+0000 7f577dffb640 10 client.?.objecter ms_handle_connect 0x7f578813f710 2024-03-25T17:31:55.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.556+0000 7f577dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:55.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.561+0000 7f577dffb640 10 client.?.objecter ms_handle_connect 0x7f57880a98f0 2024-03-25T17:31:55.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.561+0000 7f577dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:55.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.569+0000 7f577dffb640 10 client.?.objecter ms_handle_connect 0x7f57881430c0 2024-03-25T17:31:55.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.569+0000 7f577dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.572+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232576853 0 0) 0x7f57881daba0 con 0x7f578813f710 2024-03-25T17:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.572+0000 7f577dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.572+0000 7f577dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.572+0000 7f577dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:55.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.572+0000 7f577dffb640 10 monclient(hunting): my global_id is 34310 2024-03-25T17:31:55.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.572+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f576c004eb0 con 0x7f578813f710 2024-03-25T17:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.575+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700756448 0 0) 0x7f57880abc30 con 0x7f57880a98f0 2024-03-25T17:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.575+0000 7f577dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.575+0000 7f577dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.575+0000 7f577dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.575+0000 7f577dffb640 10 monclient(hunting): my global_id is 4336 2024-03-25T17:31:55.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.575+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57881daba0 con 0x7f57880a98f0 2024-03-25T17:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.576+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167870330 0 0) 0x7f57880ab9c0 con 0x7f57881430c0 2024-03-25T17:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.576+0000 7f577dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.576+0000 7f577dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.576+0000 7f577dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.576+0000 7f577dffb640 10 monclient(hunting): my global_id is 4323 2024-03-25T17:31:55.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.576+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57880abc30 con 0x7f57881430c0 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.586+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1609888620 0 0) 0x7f57881daba0 con 0x7f57880a98f0 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.586+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57880ab9c0 con 0x7f57880a98f0 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.586+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5784001ae0 con 0x7f57880a98f0 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.586+0000 7f577dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.586+0000 7f577dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.586+0000 7f577dffb640 10 monclient(hunting): dump: 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:55.587 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:55.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.586+0000 7f577dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.588+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 505687437 0 0) 0x7f576c004eb0 con 0x7f578813f710 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.588+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57881daba0 con 0x7f578813f710 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.588+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5778001a30 con 0x7f578813f710 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.588+0000 7f577dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.588+0000 7f577dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.588+0000 7f577dffb640 10 monclient(hunting): dump: 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:55.589 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:55.590 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:55.590 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:55.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 639461244 0 0) 0x7f57880ab9c0 con 0x7f57880a98f0 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 >> v1:172.21.15.134:6790/0 conn(0x7f578813f710 legacy=0x7f57881db0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 >> v1:172.21.15.134:6789/0 conn(0x7f57881430c0 legacy=0x7f5788134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 1 monclient: found mon.b 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57881db7d0 con 0x7f57880a98f0 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 10 monclient: _finish_auth 0 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.592+0000 7f577dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 5 monclient: authenticate success, global_id 4336 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 10 monclient: _renew_subs 2024-03-25T17:31:55.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5788137930 con 0x7f57880a98f0 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 10 monclient: _renew_subs 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5788134900 con 0x7f57880a98f0 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.593+0000 7f578f8e6640 1 librados: init done 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.594+0000 7f578f8e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:55.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.594+0000 7f578f8e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:55.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.594+0000 7f578f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:55.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.594+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57881dad30 con 0x7f57880a98f0 2024-03-25T17:31:55.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.887+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5784001210 con 0x7f57880a98f0 2024-03-25T17:31:55.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.887+0000 7f577dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.887+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5784003a40 con 0x7f57880a98f0 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.887+0000 7f577dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.887+0000 7f577dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.887+0000 7f577dffb640 10 monclient: dump: 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:31:55.889 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:31:55.890 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:31:55.890 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:31:55.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.887+0000 7f577f7fe640 4 set_mon_vals no callback set 2024-03-25T17:31:55.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.888+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f5784019a10 con 0x7f57880a98f0 2024-03-25T17:31:55.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.889+0000 7f578ce81640 10 monclient: get_auth_request con 0x7f576c06df70 auth_method 0 2024-03-25T17:31:55.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.889+0000 7f577dffb640 10 client.4336.objecter ms_handle_connect 0x7f576c06df70 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f5784053650 con 0x7f57880a98f0 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 10 client.4336.objecter ms_dispatch 0x7f5788000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f578404f160 con 0x7f57880a98f0 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:31:55.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:55.894+0000 7f577dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:31:56.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.121+0000 7f578f8e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:56.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.121+0000 7f578f8e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:56.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.121+0000 7f578f8e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:31:56.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.121+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57881da990 con 0x7f57880a98f0 2024-03-25T17:31:56.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.140+0000 7f577dffb640 1 -- 172.21.15.134:0/1221285464 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f5784051090 con 0x7f57880a98f0 2024-03-25T17:31:56.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.140+0000 7f577dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:31:56.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.140+0000 7f577dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:31:56.141 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:31:56.141 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:31:56.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 10 librados: watch_flush enter 2024-03-25T17:31:56.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 10 librados: watch_flush exit 2024-03-25T17:31:56.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T17:31:56.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 10 client.4336.objecter successfully canceled tick 2024-03-25T17:31:56.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 >> v1:172.21.15.134:6800/610837766 conn(0x7f576c06df70 legacy=0x7f576c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:56.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 10 monclient: shutdown 2024-03-25T17:31:56.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:31:56.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 >> v1:172.21.15.159:6789/0 conn(0x7f57880a98f0 legacy=0x7f57881386d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:31:56.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 shutdown_connections 2024-03-25T17:31:56.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 >> 172.21.15.134:0/1221285464 conn(0x7f57880b7ea0 msgr2=0x7f5788074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:31:56.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 shutdown_connections 2024-03-25T17:31:56.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 1 -- 172.21.15.134:0/1221285464 wait complete. 2024-03-25T17:31:56.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:31:56.143+0000 7f578f8e6640 1 librados: shutdown 2024-03-25T17:31:56.155 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:02.962 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:03.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.073+0000 7f4d46c63640 10 monclient: get_monmap_and_config 2024-03-25T17:32:03.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.073+0000 7f4d46c63640 10 monclient: build_initial_monmap 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.073+0000 7f4d46c63640 10 monclient: monmap: 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:03.074128+0000 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:03.074128+0000 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:03.074 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:03.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 1 Processor -- start 2024-03-25T17:32:03.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 1 -- start start 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient: init 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient(hunting): picked mon.noname-a con 0x7f4d400b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient(hunting): picked mon.noname-b con 0x7f4d400b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4014be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d400af940 con 0x7f4d400b4d60 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d400b0a70 con 0x7f4d4014be90 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d400b0d50 con 0x7f4d400b1730 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:03.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.074+0000 7f4d46c63640 10 monclient(hunting): authenticate will time out at 5476.337649s 2024-03-25T17:32:03.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.075+0000 7f4d449ff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4d400b4d60 0x7f4d4013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:57940/0 (socket says 172.21.15.134:57940) 2024-03-25T17:32:03.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.075+0000 7f4d449ff640 1 -- 172.21.15.134:0/3019939706 learned_addr learned my addr 172.21.15.134:0/3019939706 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:03.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.083+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869153545 0 0) 0x7f4d400af940 con 0x7f4d400b4d60 2024-03-25T17:32:03.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.083+0000 7f4d3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:32:03.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.083+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:03.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.083+0000 7f4d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:03.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.083+0000 7f4d3f7fe640 10 monclient(hunting): my global_id is 4326 2024-03-25T17:32:03.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.083+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30003520 con 0x7f4d400b4d60 2024-03-25T17:32:03.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4031700378 0 0) 0x7f4d30003520 con 0x7f4d400b4d60 2024-03-25T17:32:03.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 >> v1:172.21.15.134:6790/0 conn(0x7f4d400b1730 legacy=0x7f4d400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:03.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 >> v1:172.21.15.159:6789/0 conn(0x7f4d4014be90 legacy=0x7f4d4014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 1 monclient: found mon.noname-a 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d400b1030 con 0x7f4d400b4d60 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d46c63640 5 monclient: authenticate success, global_id 4326 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.087+0000 7f4d46c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.088+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d34001670 con 0x7f4d400b4d60 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.088+0000 7f4d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.088+0000 7f4d3f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:03.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.088+0000 7f4d3f7fe640 10 monclient: dump: 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:03.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.088+0000 7f4d46c63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:03.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.096+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d34001900 con 0x7f4d400b4d60 2024-03-25T17:32:03.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.096+0000 7f4d3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:03.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.096+0000 7f4d3f7fe640 1 -- 172.21.15.134:0/3019939706 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d34001c90 con 0x7f4d400b4d60 2024-03-25T17:32:03.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.096+0000 7f4d46c63640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:03.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.096+0000 7f4d46c63640 4 set_mon_vals no callback set 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.096+0000 7f4d3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.097+0000 7f4d3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.097+0000 7f4d3f7fe640 10 monclient: dump: 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:03.098 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.097+0000 7f4d46c63640 10 monclient: shutdown 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.097+0000 7f4d46c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.097+0000 7f4d46c63640 1 -- 172.21.15.134:0/3019939706 >> v1:172.21.15.134:6789/0 conn(0x7f4d400b4d60 legacy=0x7f4d4013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.098+0000 7f4d46c63640 1 -- 172.21.15.134:0/3019939706 shutdown_connections 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.098+0000 7f4d46c63640 1 -- 172.21.15.134:0/3019939706 >> 172.21.15.134:0/3019939706 conn(0x7f4d400617a0 msgr2=0x7f4d40061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.098+0000 7f4d46c63640 1 -- 172.21.15.134:0/3019939706 shutdown_connections 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.098+0000 7f4d46c63640 1 -- 172.21.15.134:0/3019939706 wait complete. 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 10 monclient: build_initial_monmap 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 10 monclient: monmap: 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:03.100 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 1 librados: starting msgr at 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 1 librados: starting objecter 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 1 Processor -- start 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 1 -- start start 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 1 librados: setting wanted keys 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 1 librados: calling monclient init 2024-03-25T17:32:03.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.099+0000 7f4d46c63640 10 monclient: init 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 10 monclient(hunting): picked mon.noname-b con 0x7f4d400b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 10 monclient(hunting): picked mon.noname-a con 0x7f4d400b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 10 monclient(hunting): picked mon.noname-c con 0x7f4d4014be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40138030 con 0x7f4d400b4d60 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40136c90 con 0x7f4d400b1730 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d40135d30 con 0x7f4d4014be90 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:03.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.100+0000 7f4d46c63640 10 monclient(hunting): authenticate will time out at 5476.363619s 2024-03-25T17:32:03.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.114+0000 7f4d45200640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f4d4014be90 0x7f4d401d7eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:47118/0 (socket says 172.21.15.134:47118) 2024-03-25T17:32:03.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.114+0000 7f4d3ffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4d400b4d60 0x7f4d401d77a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58002/0 (socket says 172.21.15.134:58002) 2024-03-25T17:32:03.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.114+0000 7f4d45200640 1 -- 172.21.15.134:0/60284132 learned_addr learned my addr 172.21.15.134:0/60284132 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:03.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.115+0000 7f4d3d7fa640 10 client.?.objecter ms_handle_connect 0x7f4d400b4d60 2024-03-25T17:32:03.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.115+0000 7f4d3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:03.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.116+0000 7f4d3d7fa640 10 client.?.objecter ms_handle_connect 0x7f4d4014be90 2024-03-25T17:32:03.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.116+0000 7f4d3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:03.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.125+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541270129 0 0) 0x7f4d40138030 con 0x7f4d400b4d60 2024-03-25T17:32:03.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.125+0000 7f4d3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:32:03.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.125+0000 7f4d3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:03.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.125+0000 7f4d3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:03.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.125+0000 7f4d3d7fa640 10 monclient(hunting): my global_id is 4329 2024-03-25T17:32:03.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.125+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d24004eb0 con 0x7f4d400b4d60 2024-03-25T17:32:03.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.129+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706528837 0 0) 0x7f4d40135d30 con 0x7f4d4014be90 2024-03-25T17:32:03.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.130+0000 7f4d3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:32:03.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.130+0000 7f4d3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:03.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.130+0000 7f4d3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:03.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.130+0000 7f4d3d7fa640 10 monclient(hunting): my global_id is 34316 2024-03-25T17:32:03.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.130+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d40138030 con 0x7f4d4014be90 2024-03-25T17:32:03.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.131+0000 7f4d3d7fa640 10 client.?.objecter ms_handle_connect 0x7f4d400b1730 2024-03-25T17:32:03.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.131+0000 7f4d3d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.132+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 654171327 0 0) 0x7f4d24004eb0 con 0x7f4d400b4d60 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.132+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d40135d30 con 0x7f4d400b4d60 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.132+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d2c001970 con 0x7f4d400b4d60 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.132+0000 7f4d3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.132+0000 7f4d3d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.132+0000 7f4d3d7fa640 10 monclient(hunting): dump: 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:03.133 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:03.134 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:03.134 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:03.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.132+0000 7f4d3d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:03.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.135+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288306387 0 0) 0x7f4d40136c90 con 0x7f4d400b1730 2024-03-25T17:32:03.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.135+0000 7f4d3d7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:32:03.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.135+0000 7f4d3d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:03.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.135+0000 7f4d3d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:03.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.135+0000 7f4d3d7fa640 10 monclient(hunting): my global_id is 4342 2024-03-25T17:32:03.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.135+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d24004eb0 con 0x7f4d400b1730 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.140+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2642105804 0 0) 0x7f4d40138030 con 0x7f4d4014be90 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.140+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d40136c90 con 0x7f4d4014be90 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.140+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d38001bf0 con 0x7f4d4014be90 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.140+0000 7f4d3d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.140+0000 7f4d3d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.140+0000 7f4d3d7fa640 10 monclient(hunting): dump: 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:03.141 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:03.142 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:03.142 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:03.142 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:03.142 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1964985202 0 0) 0x7f4d40135d30 con 0x7f4d400b4d60 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 >> v1:172.21.15.134:6790/0 conn(0x7f4d4014be90 legacy=0x7f4d401d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 >> v1:172.21.15.159:6789/0 conn(0x7f4d400b1730 legacy=0x7f4d4013b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 1 monclient: found mon.a 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d40135ec0 con 0x7f4d400b4d60 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 10 monclient: _finish_auth 0 2024-03-25T17:32:03.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d3d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 5 monclient: authenticate success, global_id 4329 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 10 monclient: _renew_subs 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4013bcf0 con 0x7f4d400b4d60 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 10 monclient: _renew_subs 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d401366b0 con 0x7f4d400b4d60 2024-03-25T17:32:03.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.142+0000 7f4d46c63640 1 librados: init done 2024-03-25T17:32:03.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.143+0000 7f4d46c63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:03.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.143+0000 7f4d46c63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:03.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.143+0000 7f4d46c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:03.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.143+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d400627b0 con 0x7f4d400b4d60 2024-03-25T17:32:03.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.357+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d2c003610 con 0x7f4d400b4d60 2024-03-25T17:32:03.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.357+0000 7f4d3d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:03.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.357+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d2c003960 con 0x7f4d400b4d60 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.357+0000 7f4d3d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.357+0000 7f4d3d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.357+0000 7f4d3d7fa640 10 monclient: dump: 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:03.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.357+0000 7f4d3effd640 4 set_mon_vals no callback set 2024-03-25T17:32:03.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.441+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f4d2c003ac0 con 0x7f4d400b4d60 2024-03-25T17:32:03.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.442+0000 7f4d449ff640 10 monclient: get_auth_request con 0x7f4d2406dfb0 auth_method 0 2024-03-25T17:32:03.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.442+0000 7f4d3d7fa640 10 client.4329.objecter ms_handle_connect 0x7f4d2406dfb0 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f4d2c0547d0 con 0x7f4d400b4d60 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 10 client.4329.objecter ms_dispatch 0x7f4d40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d2c0502e0 con 0x7f4d400b4d60 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:03.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.447+0000 7f4d3d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:03.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.674+0000 7f4d46c63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:03.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.674+0000 7f4d46c63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:03.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.674+0000 7f4d46c63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:03.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.674+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d40136e20 con 0x7f4d400b4d60 2024-03-25T17:32:03.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.702+0000 7f4d3d7fa640 1 -- 172.21.15.134:0/60284132 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f4d2c052210 con 0x7f4d400b4d60 2024-03-25T17:32:03.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.702+0000 7f4d3d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:03.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.702+0000 7f4d3d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:03.703 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:03.703 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:03.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 10 librados: watch_flush enter 2024-03-25T17:32:03.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 10 librados: watch_flush exit 2024-03-25T17:32:03.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-25T17:32:03.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 10 client.4329.objecter successfully canceled tick 2024-03-25T17:32:03.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 >> v1:172.21.15.134:6800/610837766 conn(0x7f4d2406dfb0 legacy=0x7f4d24070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:03.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 10 monclient: shutdown 2024-03-25T17:32:03.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:03.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 >> v1:172.21.15.134:6789/0 conn(0x7f4d400b4d60 legacy=0x7f4d401d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:03.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 shutdown_connections 2024-03-25T17:32:03.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 >> 172.21.15.134:0/60284132 conn(0x7f4d400617a0 msgr2=0x7f4d4013c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:03.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 shutdown_connections 2024-03-25T17:32:03.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 1 -- 172.21.15.134:0/60284132 wait complete. 2024-03-25T17:32:03.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:03.705+0000 7f4d46c63640 1 librados: shutdown 2024-03-25T17:32:03.718 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:04.371 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:09.388 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.416 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:09.444 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.417 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:09.445 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.429 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:09.446 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.444 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:09.472 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.463 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:09.480 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.479 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:09.495 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.494 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:09.510 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:04.521 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:09.537 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:10.526 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:10.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.638+0000 7f966879d640 10 monclient: get_monmap_and_config 2024-03-25T17:32:10.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.638+0000 7f966879d640 10 monclient: build_initial_monmap 2024-03-25T17:32:10.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.638+0000 7f966879d640 10 monclient: monmap: 2024-03-25T17:32:10.639 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:10.639 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:10.639 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:10.639597+0000 2024-03-25T17:32:10.640 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:10.639597+0000 2024-03-25T17:32:10.640 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:10.640 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:10.640 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:10.640 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:10.640 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:10.640 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.639+0000 7f966879d640 1 Processor -- start 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.639+0000 7f966879d640 1 -- start start 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.639+0000 7f966879d640 10 monclient: init 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 10 monclient(hunting): picked mon.noname-b con 0x7f96600a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 10 monclient(hunting): picked mon.noname-c con 0x7f966013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 10 monclient(hunting): picked mon.noname-a con 0x7f96601430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9660142520 con 0x7f96601430c0 2024-03-25T17:32:10.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96600b28a0 con 0x7f966013f710 2024-03-25T17:32:10.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96600b2a30 con 0x7f96600a98f0 2024-03-25T17:32:10.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:10.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.640+0000 7f966879d640 10 monclient(hunting): authenticate will time out at 5483.903066s 2024-03-25T17:32:10.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.649+0000 7f9666d3a640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f96601430c0 0x7f966014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58542/0 (socket says 172.21.15.134:58542) 2024-03-25T17:32:10.649 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.649+0000 7f9666d3a640 1 -- 172.21.15.134:0/3724321650 learned_addr learned my addr 172.21.15.134:0/3724321650 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:10.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.662+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163968626 0 0) 0x7f9660142520 con 0x7f96601430c0 2024-03-25T17:32:10.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.662+0000 7f9665537640 10 monclient(hunting): _negotiate 2024-03-25T17:32:10.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.662+0000 7f9665537640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:10.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.662+0000 7f9665537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:10.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.662+0000 7f9665537640 10 monclient(hunting): my global_id is 4332 2024-03-25T17:32:10.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.662+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f964c003520 con 0x7f96601430c0 2024-03-25T17:32:10.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.664+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792641783 0 0) 0x7f96600b28a0 con 0x7f966013f710 2024-03-25T17:32:10.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.664+0000 7f9665537640 10 monclient(hunting): _negotiate 2024-03-25T17:32:10.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.664+0000 7f9665537640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:10.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.664+0000 7f9665537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:10.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.664+0000 7f9665537640 10 monclient(hunting): my global_id is 4345 2024-03-25T17:32:10.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.664+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9660142520 con 0x7f966013f710 2024-03-25T17:32:10.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.668+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2408196319 0 0) 0x7f96600b2a30 con 0x7f96600a98f0 2024-03-25T17:32:10.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.668+0000 7f9665537640 10 monclient(hunting): _negotiate 2024-03-25T17:32:10.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.668+0000 7f9665537640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:10.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.668+0000 7f9665537640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:10.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.668+0000 7f9665537640 10 monclient(hunting): my global_id is 34319 2024-03-25T17:32:10.669 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.668+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96600b28a0 con 0x7f96600a98f0 2024-03-25T17:32:10.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444343336 0 0) 0x7f964c003520 con 0x7f96601430c0 2024-03-25T17:32:10.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 >> v1:172.21.15.134:6790/0 conn(0x7f96600a98f0 legacy=0x7f96600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:10.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 >> v1:172.21.15.159:6789/0 conn(0x7f966013f710 legacy=0x7f96601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 10 monclient: _finish_hunting 0 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 1 monclient: found mon.noname-a 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96600b44c0 con 0x7f96601430c0 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 10 monclient: _finish_auth 0 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 10 monclient: _check_auth_tickets 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:10.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9658001a10 con 0x7f96601430c0 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f9665537640 10 monclient: dump: 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:10.674 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:10.675 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:10.675 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:10.675 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:10.675 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:10.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f966879d640 5 monclient: authenticate success, global_id 4332 2024-03-25T17:32:10.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.671+0000 7f966879d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:10.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9658001b70 con 0x7f96601430c0 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f9665537640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f9665537640 1 -- 172.21.15.134:0/3724321650 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9658001f80 con 0x7f96601430c0 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f966879d640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f966879d640 4 set_mon_vals no callback set 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f9665537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f9665537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.680+0000 7f9665537640 10 monclient: dump: 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:10.681 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:10.682 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:10.682 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:10.682 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:10.682 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.681+0000 7f966879d640 10 monclient: shutdown 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.681+0000 7f966879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.681+0000 7f966879d640 1 -- 172.21.15.134:0/3724321650 >> v1:172.21.15.134:6789/0 conn(0x7f96601430c0 legacy=0x7f966014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.681+0000 7f966879d640 1 -- 172.21.15.134:0/3724321650 shutdown_connections 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.681+0000 7f966879d640 1 -- 172.21.15.134:0/3724321650 >> 172.21.15.134:0/3724321650 conn(0x7f96600b7ea0 msgr2=0x7f96600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.681+0000 7f966879d640 1 -- 172.21.15.134:0/3724321650 shutdown_connections 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.681+0000 7f966879d640 1 -- 172.21.15.134:0/3724321650 wait complete. 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 10 monclient: build_initial_monmap 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 10 monclient: monmap: 2024-03-25T17:32:10.684 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 1 librados: starting msgr at 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 1 librados: starting objecter 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 1 Processor -- start 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 1 -- start start 2024-03-25T17:32:10.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 1 librados: setting wanted keys 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 1 librados: calling monclient init 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.682+0000 7f966879d640 10 monclient: init 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 10 monclient(hunting): picked mon.noname-a con 0x7f96600a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 10 monclient(hunting): picked mon.noname-b con 0x7f966013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 10 monclient(hunting): picked mon.noname-c con 0x7f96601430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96600b7c30 con 0x7f96600a98f0 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9660137520 con 0x7f966013f710 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96601d7ba0 con 0x7f96601430c0 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:10.686 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.683+0000 7f966879d640 10 monclient(hunting): authenticate will time out at 5483.946088s 2024-03-25T17:32:10.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.703+0000 7f9666d3a640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f96601430c0 0x7f96601e2cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59532/0 (socket says 172.21.15.134:59532) 2024-03-25T17:32:10.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.703+0000 7f9666d3a640 1 -- 172.21.15.134:0/2612419475 learned_addr learned my addr 172.21.15.134:0/2612419475 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:10.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.704+0000 7f9656ffd640 10 client.?.objecter ms_handle_connect 0x7f966013f710 2024-03-25T17:32:10.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.704+0000 7f9656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:10.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.712+0000 7f9656ffd640 10 client.?.objecter ms_handle_connect 0x7f96601430c0 2024-03-25T17:32:10.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.712+0000 7f9656ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:10.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.713+0000 7f9666539640 1 -- 172.21.15.134:0/2612419475 >> v1:172.21.15.134:6789/0 conn(0x7f96600a98f0 legacy=0x7f96601d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T17:32:10.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.713+0000 7f9666539640 1 -- 172.21.15.134:0/2612419475 >> v1:172.21.15.134:6789/0 conn(0x7f96600a98f0 legacy=0x7f96601d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T17:32:10.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.713+0000 7f9666539640 1 --1- 172.21.15.134:0/2612419475 >> v1:172.21.15.134:6789/0 conn(0x7f96600a98f0 0x7f96601d7490 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T17:32:10.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.713+0000 7f9666539640 1 --1- 172.21.15.134:0/2612419475 >> v1:172.21.15.134:6789/0 conn(0x7f96600a98f0 0x7f96601d7490 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T17:32:10.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.713+0000 7f9656ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.134:6789/0 2024-03-25T17:32:10.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.718+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808732559 0 0) 0x7f9660137520 con 0x7f966013f710 2024-03-25T17:32:10.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.718+0000 7f9656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:10.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.718+0000 7f9656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:10.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.718+0000 7f9656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:10.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.718+0000 7f9656ffd640 10 monclient(hunting): my global_id is 4348 2024-03-25T17:32:10.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.718+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9644004eb0 con 0x7f966013f710 2024-03-25T17:32:10.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.729+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2327142710 0 0) 0x7f9644004eb0 con 0x7f966013f710 2024-03-25T17:32:10.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.729+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9660137520 con 0x7f966013f710 2024-03-25T17:32:10.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.729+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9650001970 con 0x7f966013f710 2024-03-25T17:32:10.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.729+0000 7f9656ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:10.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.729+0000 7f9656ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:10.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.729+0000 7f9656ffd640 10 monclient(hunting): dump: 2024-03-25T17:32:10.730 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:10.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.729+0000 7f9656ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:10.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.735+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173868801 0 0) 0x7f96601d7ba0 con 0x7f96601430c0 2024-03-25T17:32:10.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.735+0000 7f9656ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:10.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.735+0000 7f9656ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:10.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.735+0000 7f9656ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:10.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.735+0000 7f9656ffd640 10 monclient(hunting): my global_id is 34322 2024-03-25T17:32:10.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.735+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9644004eb0 con 0x7f96601430c0 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 702767851 0 0) 0x7f9660137520 con 0x7f966013f710 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 >> v1:172.21.15.134:6790/0 conn(0x7f96601430c0 legacy=0x7f96601e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 >> v1:172.21.15.134:6789/0 conn(0x7f96600a98f0 legacy=0x7f96601d7490 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 1 monclient: found mon.b 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9660136950 con 0x7f966013f710 2024-03-25T17:32:10.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 10 monclient: _finish_auth 0 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f9656ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 5 monclient: authenticate success, global_id 4348 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 10 monclient: _renew_subs 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9660136ba0 con 0x7f966013f710 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 10 monclient: _renew_subs 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9660137010 con 0x7f966013f710 2024-03-25T17:32:10.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.752+0000 7f966879d640 1 librados: init done 2024-03-25T17:32:10.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.753+0000 7f966879d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:10.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.753+0000 7f966879d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:10.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.753+0000 7f966879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:10.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:10.753+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9660137700 con 0x7f966013f710 2024-03-25T17:32:11.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.042+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9650003610 con 0x7f966013f710 2024-03-25T17:32:11.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.042+0000 7f9656ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:11.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.042+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9650003960 con 0x7f966013f710 2024-03-25T17:32:11.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.042+0000 7f9656ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:11.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.042+0000 7f9656ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:11.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.042+0000 7f9656ffd640 10 monclient: dump: 2024-03-25T17:32:11.043 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:11.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.042+0000 7f9664d36640 4 set_mon_vals no callback set 2024-03-25T17:32:11.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.043+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f96500199f0 con 0x7f966013f710 2024-03-25T17:32:11.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.043+0000 7f9666539640 10 monclient: get_auth_request con 0x7f964406dd70 auth_method 0 2024-03-25T17:32:11.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.044+0000 7f9656ffd640 10 client.4348.objecter ms_handle_connect 0x7f964406dd70 2024-03-25T17:32:11.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.049+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f9650053610 con 0x7f966013f710 2024-03-25T17:32:11.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.049+0000 7f9656ffd640 10 client.4348.objecter ms_dispatch 0x7f9660000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:11.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.049+0000 7f9656ffd640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:11.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.049+0000 7f9656ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:11.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.049+0000 7f9656ffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-25T17:32:11.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.050+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f965004ef50 con 0x7f966013f710 2024-03-25T17:32:11.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.050+0000 7f9656ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:11.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.050+0000 7f9656ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:11.275 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.275+0000 7f966879d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:11.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.275+0000 7f966879d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:11.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.275+0000 7f966879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:11.276 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.275+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96600627b0 con 0x7f966013f710 2024-03-25T17:32:11.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.315+0000 7f9656ffd640 1 -- 172.21.15.134:0/2612419475 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f9650051050 con 0x7f966013f710 2024-03-25T17:32:11.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.316+0000 7f9656ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:11.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.316+0000 7f9656ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:11.316 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:11.317 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:11.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 10 librados: watch_flush enter 2024-03-25T17:32:11.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 10 librados: watch_flush exit 2024-03-25T17:32:11.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 10 client.4348.objecter successfully canceled tick 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 >> v1:172.21.15.134:6800/610837766 conn(0x7f964406dd70 legacy=0x7f9644070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 10 monclient: shutdown 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 >> v1:172.21.15.159:6789/0 conn(0x7f966013f710 legacy=0x7f9660133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 shutdown_connections 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 >> 172.21.15.134:0/2612419475 conn(0x7f96600b7ea0 msgr2=0x7f966013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 shutdown_connections 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 1 -- 172.21.15.134:0/2612419475 wait complete. 2024-03-25T17:32:11.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:11.318+0000 7f966879d640 1 librados: shutdown 2024-03-25T17:32:11.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:18.138 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.253+0000 7f08dd35d640 10 monclient: get_monmap_and_config 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.253+0000 7f08dd35d640 10 monclient: build_initial_monmap 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.253+0000 7f08dd35d640 10 monclient: monmap: 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:18.255712+0000 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:18.255712+0000 2024-03-25T17:32:18.255 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:18.256 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:18.256 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:18.256 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:18.256 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:18.256 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.254+0000 7f08dd35d640 1 Processor -- start 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.254+0000 7f08dd35d640 1 -- start start 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.254+0000 7f08dd35d640 10 monclient: init 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 10 monclient(hunting): picked mon.noname-a con 0x7f08d80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 10 monclient(hunting): picked mon.noname-b con 0x7f08d813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 10 monclient(hunting): picked mon.noname-c con 0x7f08d81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d8142520 con 0x7f08d80a98f0 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80b28a0 con 0x7f08d81430c0 2024-03-25T17:32:18.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d80b2a30 con 0x7f08d813f710 2024-03-25T17:32:18.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:18.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.255+0000 7f08dd35d640 10 monclient(hunting): authenticate will time out at 5491.519333s 2024-03-25T17:32:18.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.266+0000 7f08cffff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f08d813f710 0x7f08d81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:36052/0 (socket says 172.21.15.134:36052) 2024-03-25T17:32:18.267 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.266+0000 7f08cffff640 1 -- 172.21.15.134:0/3751811125 learned_addr learned my addr 172.21.15.134:0/3751811125 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:18.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.278+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079129583 0 0) 0x7f08d80b28a0 con 0x7f08d81430c0 2024-03-25T17:32:18.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.278+0000 7f08d67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:18.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.278+0000 7f08d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:18.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.278+0000 7f08d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:18.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.278+0000 7f08d67fc640 10 monclient(hunting): my global_id is 4351 2024-03-25T17:32:18.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.278+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08c4003520 con 0x7f08d81430c0 2024-03-25T17:32:18.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.281+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3167861354 0 0) 0x7f08d80b2a30 con 0x7f08d813f710 2024-03-25T17:32:18.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.281+0000 7f08d67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:18.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.281+0000 7f08d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:18.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.281+0000 7f08d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:18.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.281+0000 7f08d67fc640 10 monclient(hunting): my global_id is 34325 2024-03-25T17:32:18.283 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.281+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08d80b28a0 con 0x7f08d813f710 2024-03-25T17:32:18.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410484485 0 0) 0x7f08c4003520 con 0x7f08d81430c0 2024-03-25T17:32:18.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 >> v1:172.21.15.134:6790/0 conn(0x7f08d813f710 legacy=0x7f08d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:18.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 >> v1:172.21.15.134:6789/0 conn(0x7f08d80a98f0 legacy=0x7f08d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:18.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 10 monclient: _finish_hunting 0 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 1 monclient: found mon.noname-c 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d80b44c0 con 0x7f08d81430c0 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 10 monclient: _finish_auth 0 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 10 monclient: _check_auth_tickets 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f08d0001a10 con 0x7f08d81430c0 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08d67fc640 10 monclient: dump: 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.295 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08dd35d640 5 monclient: authenticate success, global_id 4351 2024-03-25T17:32:18.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.292+0000 7f08dd35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08d0001b70 con 0x7f08d81430c0 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08d67fc640 1 -- 172.21.15.134:0/3751811125 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f08d0001f80 con 0x7f08d81430c0 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08d67fc640 10 monclient: dump: 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.318 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08dd35d640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:18.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.316+0000 7f08dd35d640 4 set_mon_vals no callback set 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.317+0000 7f08dd35d640 10 monclient: shutdown 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.317+0000 7f08dd35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.317+0000 7f08dd35d640 1 -- 172.21.15.134:0/3751811125 >> v1:172.21.15.159:6789/0 conn(0x7f08d81430c0 legacy=0x7f08d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.317+0000 7f08dd35d640 1 -- 172.21.15.134:0/3751811125 shutdown_connections 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.317+0000 7f08dd35d640 1 -- 172.21.15.134:0/3751811125 >> 172.21.15.134:0/3751811125 conn(0x7f08d80b7ea0 msgr2=0x7f08d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.317+0000 7f08dd35d640 1 -- 172.21.15.134:0/3751811125 shutdown_connections 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.317+0000 7f08dd35d640 1 -- 172.21.15.134:0/3751811125 wait complete. 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 10 monclient: build_initial_monmap 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 10 monclient: monmap: 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:18.321 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 1 librados: starting msgr at 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 1 librados: starting objecter 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 1 Processor -- start 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 1 -- start start 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 1 librados: setting wanted keys 2024-03-25T17:32:18.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 1 librados: calling monclient init 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.318+0000 7f08dd35d640 10 monclient: init 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 10 monclient(hunting): picked mon.noname-c con 0x7f08d80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 10 monclient(hunting): picked mon.noname-a con 0x7f08d813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 10 monclient(hunting): picked mon.noname-b con 0x7f08d81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d81354c0 con 0x7f08d813f710 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d81319b0 con 0x7f08d81430c0 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d8135070 con 0x7f08d80a98f0 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:18.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.319+0000 7f08dd35d640 10 monclient(hunting): authenticate will time out at 5491.583087s 2024-03-25T17:32:18.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.325+0000 7f08d77fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f08d81430c0 0x7f08d81312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:38460/0 (socket says 172.21.15.134:38460) 2024-03-25T17:32:18.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.325+0000 7f08d77fe640 1 -- 172.21.15.134:0/3503185420 learned_addr learned my addr 172.21.15.134:0/3503185420 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:18.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.326+0000 7f08cf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d81430c0 2024-03-25T17:32:18.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.326+0000 7f08cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:18.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.339+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609988790 0 0) 0x7f08d81319b0 con 0x7f08d81430c0 2024-03-25T17:32:18.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.339+0000 7f08cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:32:18.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.339+0000 7f08cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:18.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.339+0000 7f08cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:18.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.339+0000 7f08cf7fe640 10 monclient(hunting): my global_id is 4354 2024-03-25T17:32:18.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.339+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08b8004eb0 con 0x7f08d81430c0 2024-03-25T17:32:18.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.341+0000 7f08cf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d813f710 2024-03-25T17:32:18.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.341+0000 7f08cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:18.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.348+0000 7f08cf7fe640 10 client.?.objecter ms_handle_connect 0x7f08d80a98f0 2024-03-25T17:32:18.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.348+0000 7f08cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.351+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1417441235 0 0) 0x7f08b8004eb0 con 0x7f08d81430c0 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.351+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f08d81319b0 con 0x7f08d81430c0 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.351+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f08d0001f40 con 0x7f08d81430c0 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.351+0000 7f08cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.351+0000 7f08cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.351+0000 7f08cf7fe640 10 monclient(hunting): dump: 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:18.353 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:18.354 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:18.354 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:18.354 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:18.354 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:18.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.351+0000 7f08cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2012527131 0 0) 0x7f08d81319b0 con 0x7f08d81430c0 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 >> v1:172.21.15.134:6790/0 conn(0x7f08d80a98f0 legacy=0x7f08d81d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 >> v1:172.21.15.134:6789/0 conn(0x7f08d813f710 legacy=0x7f08d8130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 1 monclient: found mon.b 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:18.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d8135200 con 0x7f08d81430c0 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 10 monclient: _finish_auth 0 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 5 monclient: authenticate success, global_id 4354 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 10 monclient: _renew_subs 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d80b7010 con 0x7f08d81430c0 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 10 monclient: _renew_subs 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d813ba70 con 0x7f08d81430c0 2024-03-25T17:32:18.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.367+0000 7f08dd35d640 1 librados: init done 2024-03-25T17:32:18.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.368+0000 7f08dd35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:18.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.368+0000 7f08dd35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:18.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.368+0000 7f08dd35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:18.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.368+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d80b73b0 con 0x7f08d81430c0 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.651+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08d0001510 con 0x7f08d81430c0 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.651+0000 7f08cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.651+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f08d0003dc0 con 0x7f08d81430c0 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.651+0000 7f08cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.651+0000 7f08cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.651+0000 7f08cf7fe640 10 monclient: dump: 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:18.653 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:18.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.651+0000 7f08d5ffb640 4 set_mon_vals no callback set 2024-03-25T17:32:18.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.652+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f08d0003fa0 con 0x7f08d81430c0 2024-03-25T17:32:18.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.652+0000 7f08d6ffd640 10 monclient: get_auth_request con 0x7f08b806dd90 auth_method 0 2024-03-25T17:32:18.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.653+0000 7f08cf7fe640 10 client.4354.objecter ms_handle_connect 0x7f08b806dd90 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.657+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f08d0054b20 con 0x7f08d81430c0 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.657+0000 7f08cf7fe640 10 client.4354.objecter ms_dispatch 0x7f08d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.657+0000 7f08cf7fe640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.657+0000 7f08cf7fe640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.657+0000 7f08cf7fe640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.658+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08d0050460 con 0x7f08d81430c0 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.658+0000 7f08cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:18.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.658+0000 7f08cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:18.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.873+0000 7f08dd35d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:18.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.873+0000 7f08dd35d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:18.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.873+0000 7f08dd35d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:18.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.873+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08d8135670 con 0x7f08d81430c0 2024-03-25T17:32:18.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.894+0000 7f08cf7fe640 1 -- 172.21.15.134:0/3503185420 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f08d8135670 con 0x7f08d81430c0 2024-03-25T17:32:18.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.894+0000 7f08cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:18.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.894+0000 7f08cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:18.896 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:18.896 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:18.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 10 librados: watch_flush enter 2024-03-25T17:32:18.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 10 librados: watch_flush exit 2024-03-25T17:32:18.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T17:32:18.899 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 10 client.4354.objecter successfully canceled tick 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 >> v1:172.21.15.134:6800/610837766 conn(0x7f08b806dd90 legacy=0x7f08b8070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 10 monclient: shutdown 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 >> v1:172.21.15.159:6789/0 conn(0x7f08d81430c0 legacy=0x7f08d81312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 shutdown_connections 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 >> 172.21.15.134:0/3503185420 conn(0x7f08d80b7ea0 msgr2=0x7f08d813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 shutdown_connections 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 1 -- 172.21.15.134:0/3503185420 wait complete. 2024-03-25T17:32:18.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:18.897+0000 7f08dd35d640 1 librados: shutdown 2024-03-25T17:32:18.913 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:24.444 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:29.460 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.489 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:29.518 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.505 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:29.533 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.505 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:29.521 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.533 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:29.550 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.539 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:29.568 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.566 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:29.582 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.572 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:29.588 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:24.601 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:29.618 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:25.720 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:25.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.833+0000 7f602fccf640 10 monclient: get_monmap_and_config 2024-03-25T17:32:25.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.833+0000 7f602fccf640 10 monclient: build_initial_monmap 2024-03-25T17:32:25.833 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.833+0000 7f602fccf640 10 monclient: monmap: 2024-03-25T17:32:25.833 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:25.834112+0000 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:25.834112+0000 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:25.834 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:25.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.833+0000 7f602fccf640 1 Processor -- start 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.834+0000 7f602fccf640 1 -- start start 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.834+0000 7f602fccf640 10 monclient: init 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 10 monclient(hunting): picked mon.noname-a con 0x7f60280a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 10 monclient(hunting): picked mon.noname-c con 0x7f602813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 10 monclient(hunting): picked mon.noname-b con 0x7f60281430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6028142520 con 0x7f60280a98f0 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60280b28a0 con 0x7f602813f710 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60280b2a30 con 0x7f60281430c0 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:25.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.835+0000 7f602fccf640 10 monclient(hunting): authenticate will time out at 5499.097988s 2024-03-25T17:32:25.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.843+0000 7f602d26a640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f602813f710 0x7f60281429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:52658/0 (socket says 172.21.15.134:52658) 2024-03-25T17:32:25.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.843+0000 7f602d26a640 1 -- 172.21.15.134:0/2792857786 learned_addr learned my addr 172.21.15.134:0/2792857786 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:25.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.848+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 303446812 0 0) 0x7f60280b28a0 con 0x7f602813f710 2024-03-25T17:32:25.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.848+0000 7f602ca69640 10 monclient(hunting): _negotiate 2024-03-25T17:32:25.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.848+0000 7f602ca69640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:25.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.848+0000 7f602ca69640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:25.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.848+0000 7f602ca69640 10 monclient(hunting): my global_id is 4357 2024-03-25T17:32:25.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.848+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6014003520 con 0x7f602813f710 2024-03-25T17:32:25.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1413386553 0 0) 0x7f6014003520 con 0x7f602813f710 2024-03-25T17:32:25.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 >> v1:172.21.15.134:6790/0 conn(0x7f60281430c0 legacy=0x7f602814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 >> v1:172.21.15.134:6789/0 conn(0x7f60280a98f0 legacy=0x7f60280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 10 monclient: _finish_hunting 0 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 1 monclient: found mon.noname-c 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280b44c0 con 0x7f602813f710 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 10 monclient: _finish_auth 0 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 10 monclient: _check_auth_tickets 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602ca69640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602fccf640 5 monclient: authenticate success, global_id 4357 2024-03-25T17:32:25.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.854+0000 7f602fccf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6018001690 con 0x7f602813f710 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 10 monclient: dump: 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:25.867 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6018001960 con 0x7f602813f710 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 1 -- 172.21.15.134:0/2792857786 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6018001cf0 con 0x7f602813f710 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602ca69640 10 monclient: dump: 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:25.868 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602fccf640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:25.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.866+0000 7f602fccf640 4 set_mon_vals no callback set 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.867+0000 7f602fccf640 10 monclient: shutdown 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.867+0000 7f602fccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.867+0000 7f602fccf640 1 -- 172.21.15.134:0/2792857786 >> v1:172.21.15.159:6789/0 conn(0x7f602813f710 legacy=0x7f60281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.867+0000 7f602fccf640 1 -- 172.21.15.134:0/2792857786 shutdown_connections 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.867+0000 7f602fccf640 1 -- 172.21.15.134:0/2792857786 >> 172.21.15.134:0/2792857786 conn(0x7f60280b7ea0 msgr2=0x7f60280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.867+0000 7f602fccf640 1 -- 172.21.15.134:0/2792857786 shutdown_connections 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.867+0000 7f602fccf640 1 -- 172.21.15.134:0/2792857786 wait complete. 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.868+0000 7f602fccf640 10 monclient: build_initial_monmap 2024-03-25T17:32:25.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.868+0000 7f602fccf640 10 monclient: monmap: 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.868+0000 7f602fccf640 1 librados: starting msgr at 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.868+0000 7f602fccf640 1 librados: starting objecter 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 1 Processor -- start 2024-03-25T17:32:25.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 1 -- start start 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 1 librados: setting wanted keys 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 1 librados: calling monclient init 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 10 monclient: init 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 10 monclient(hunting): picked mon.noname-a con 0x7f60280a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 10 monclient(hunting): picked mon.noname-b con 0x7f602813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 10 monclient(hunting): picked mon.noname-c con 0x7f60281430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60280ab900 con 0x7f60280a98f0 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.869+0000 7f602fccf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6028137850 con 0x7f602813f710 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.870+0000 7f602fccf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60281351e0 con 0x7f60281430c0 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.870+0000 7f602fccf640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:25.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.870+0000 7f602fccf640 10 monclient(hunting): authenticate will time out at 5499.132791s 2024-03-25T17:32:25.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.879+0000 7f602da6b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f60280a98f0 0x7f6028135700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:57990/0 (socket says 172.21.15.134:57990) 2024-03-25T17:32:25.880 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.879+0000 7f602da6b640 1 -- 172.21.15.134:0/4179432482 learned_addr learned my addr 172.21.15.134:0/4179432482 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:25.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.887+0000 7f601e7fc640 10 client.?.objecter ms_handle_connect 0x7f60280a98f0 2024-03-25T17:32:25.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.887+0000 7f601e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:25.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.889+0000 7f601e7fc640 10 client.?.objecter ms_handle_connect 0x7f60281430c0 2024-03-25T17:32:25.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.889+0000 7f601e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:25.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.893+0000 7f601e7fc640 10 client.?.objecter ms_handle_connect 0x7f602813f710 2024-03-25T17:32:25.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.893+0000 7f601e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:25.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.900+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 656130854 0 0) 0x7f60280ab900 con 0x7f60280a98f0 2024-03-25T17:32:25.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.900+0000 7f601e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:25.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.900+0000 7f601e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:25.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.900+0000 7f601e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:25.900 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.900+0000 7f601e7fc640 10 monclient(hunting): my global_id is 4341 2024-03-25T17:32:25.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.900+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f600c004eb0 con 0x7f60280a98f0 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.902+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187704890 0 0) 0x7f60281351e0 con 0x7f60281430c0 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.902+0000 7f601e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.902+0000 7f601e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.902+0000 7f601e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.902+0000 7f601e7fc640 10 monclient(hunting): my global_id is 34331 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.902+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60280ab900 con 0x7f60281430c0 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.903+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18644204 0 0) 0x7f6028137850 con 0x7f602813f710 2024-03-25T17:32:25.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.903+0000 7f601e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:25.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.903+0000 7f601e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:25.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.903+0000 7f601e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:25.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.903+0000 7f601e7fc640 10 monclient(hunting): my global_id is 4360 2024-03-25T17:32:25.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.903+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60281351e0 con 0x7f602813f710 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.914+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3179927876 0 0) 0x7f600c004eb0 con 0x7f60280a98f0 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.914+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6028137850 con 0x7f60280a98f0 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.914+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6024001ce0 con 0x7f60280a98f0 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.914+0000 7f601e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.914+0000 7f601e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.914+0000 7f601e7fc640 10 monclient(hunting): dump: 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:25.915 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:25.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.914+0000 7f601e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:25.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.921+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 146603776 0 0) 0x7f6028137850 con 0x7f60280a98f0 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.921+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 >> v1:172.21.15.134:6790/0 conn(0x7f60281430c0 legacy=0x7f60281e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.921+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 >> v1:172.21.15.159:6789/0 conn(0x7f602813f710 legacy=0x7f6028135e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 1 monclient: found mon.a 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6028134e70 con 0x7f60280a98f0 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f601e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 5 monclient: authenticate success, global_id 4341 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 10 monclient: _renew_subs 2024-03-25T17:32:25.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:25.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6028138c00 con 0x7f60280a98f0 2024-03-25T17:32:25.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:25.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 10 monclient: _renew_subs 2024-03-25T17:32:25.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:25.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60281387d0 con 0x7f60280a98f0 2024-03-25T17:32:25.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.922+0000 7f602fccf640 1 librados: init done 2024-03-25T17:32:25.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.923+0000 7f602fccf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:25.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.923+0000 7f602fccf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:25.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.923+0000 7f602fccf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:25.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.923+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6028135370 con 0x7f60280a98f0 2024-03-25T17:32:25.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.929+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6024003970 con 0x7f60280a98f0 2024-03-25T17:32:25.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.929+0000 7f601e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:25.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:25.929+0000 7f601ffff640 4 set_mon_vals no callback set 2024-03-25T17:32:26.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.217+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6024003cc0 con 0x7f60280a98f0 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.217+0000 7f601e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.217+0000 7f601e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.217+0000 7f601e7fc640 10 monclient: dump: 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:26.218 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:26.219 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:26.219 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:26.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.218+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f6024003ea0 con 0x7f60280a98f0 2024-03-25T17:32:26.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.218+0000 7f602d26a640 10 monclient: get_auth_request con 0x7f600c06dda0 auth_method 0 2024-03-25T17:32:26.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.219+0000 7f601e7fc640 10 client.4341.objecter ms_handle_connect 0x7f600c06dda0 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f6024054bb0 con 0x7f60280a98f0 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 10 client.4341.objecter ms_dispatch 0x7f6028000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60240506c0 con 0x7f60280a98f0 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:26.227 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.226+0000 7f601e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:26.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.457+0000 7f602fccf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:26.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.457+0000 7f602fccf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:26.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.457+0000 7f602fccf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:32:26.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.457+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60280627b0 con 0x7f60280a98f0 2024-03-25T17:32:26.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.484+0000 7f601e7fc640 1 -- 172.21.15.134:0/4179432482 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f60240525f0 con 0x7f60280a98f0 2024-03-25T17:32:26.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.484+0000 7f601e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:26.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.484+0000 7f601e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:26.485 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:26.485 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 10 librados: watch_flush enter 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 10 librados: watch_flush exit 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 10 client.4341.objecter successfully canceled tick 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 >> v1:172.21.15.134:6800/610837766 conn(0x7f600c06dda0 legacy=0x7f600c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 10 monclient: shutdown 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 >> v1:172.21.15.134:6789/0 conn(0x7f60280a98f0 legacy=0x7f6028135700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:26.488 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 shutdown_connections 2024-03-25T17:32:26.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 >> 172.21.15.134:0/4179432482 conn(0x7f60280b7ea0 msgr2=0x7f6028074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:26.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 shutdown_connections 2024-03-25T17:32:26.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 1 -- 172.21.15.134:0/4179432482 wait complete. 2024-03-25T17:32:26.489 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:26.487+0000 7f602fccf640 1 librados: shutdown 2024-03-25T17:32:26.500 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:33.307 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:33.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.419+0000 7fc8708fb640 10 monclient: get_monmap_and_config 2024-03-25T17:32:33.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.419+0000 7fc8708fb640 10 monclient: build_initial_monmap 2024-03-25T17:32:33.419 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.419+0000 7fc8708fb640 10 monclient: monmap: 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:33.420105+0000 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:33.420105+0000 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:33.420 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:33.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.419+0000 7fc8708fb640 1 Processor -- start 2024-03-25T17:32:33.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.419+0000 7fc8708fb640 1 -- start start 2024-03-25T17:32:33.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.419+0000 7fc8708fb640 10 monclient: init 2024-03-25T17:32:33.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.420+0000 7fc8708fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:33.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.420+0000 7fc8708fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:32:33.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.420+0000 7fc8708fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:33.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.420+0000 7fc8708fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:33.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.420+0000 7fc8708fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:33.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.420+0000 7fc8708fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c142520 con 0x7fc86c0a98f0 2024-03-25T17:32:33.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.420+0000 7fc8708fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0b28a0 con 0x7fc86c1430c0 2024-03-25T17:32:33.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.421+0000 7fc8708fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0b2a30 con 0x7fc86c13f710 2024-03-25T17:32:33.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.421+0000 7fc8708fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:33.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.421+0000 7fc8708fb640 10 monclient(hunting): authenticate will time out at 5506.683775s 2024-03-25T17:32:33.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.427+0000 7fc869d9b640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc86c13f710 0x7fc86c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:54062/0 (socket says 172.21.15.134:54062) 2024-03-25T17:32:33.428 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.427+0000 7fc869d9b640 1 -- 172.21.15.134:0/3620164833 learned_addr learned my addr 172.21.15.134:0/3620164833 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:33.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.446+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074862448 0 0) 0x7fc86c0b28a0 con 0x7fc86c1430c0 2024-03-25T17:32:33.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.446+0000 7fc86959a640 10 monclient(hunting): _negotiate 2024-03-25T17:32:33.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.446+0000 7fc86959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:33.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.446+0000 7fc86959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:33.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.446+0000 7fc86959a640 10 monclient(hunting): my global_id is 4363 2024-03-25T17:32:33.447 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.446+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc854003520 con 0x7fc86c1430c0 2024-03-25T17:32:33.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.447+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196268386 0 0) 0x7fc86c0b2a30 con 0x7fc86c13f710 2024-03-25T17:32:33.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.447+0000 7fc86959a640 10 monclient(hunting): _negotiate 2024-03-25T17:32:33.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.447+0000 7fc86959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:33.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.447+0000 7fc86959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:33.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.447+0000 7fc86959a640 10 monclient(hunting): my global_id is 34334 2024-03-25T17:32:33.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.447+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c0b28a0 con 0x7fc86c13f710 2024-03-25T17:32:33.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.448+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133928177 0 0) 0x7fc86c142520 con 0x7fc86c0a98f0 2024-03-25T17:32:33.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.448+0000 7fc86959a640 10 monclient(hunting): _negotiate 2024-03-25T17:32:33.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.448+0000 7fc86959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:33.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.448+0000 7fc86959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:33.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.448+0000 7fc86959a640 10 monclient(hunting): my global_id is 4344 2024-03-25T17:32:33.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.448+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c0b2a30 con 0x7fc86c0a98f0 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.454+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 161740784 0 0) 0x7fc86c0b2a30 con 0x7fc86c0a98f0 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 >> v1:172.21.15.134:6790/0 conn(0x7fc86c13f710 legacy=0x7fc86c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 >> v1:172.21.15.159:6789/0 conn(0x7fc86c1430c0 legacy=0x7fc86c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 10 monclient: _finish_hunting 0 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 1 monclient: found mon.noname-a 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0b44c0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 10 monclient: _finish_auth 0 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 10 monclient: _check_auth_tickets 2024-03-25T17:32:33.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc8600015b0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc86959a640 10 monclient: dump: 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:33.457 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:33.458 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:33.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc8708fb640 5 monclient: authenticate success, global_id 4344 2024-03-25T17:32:33.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.455+0000 7fc8708fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:33.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.458+0000 7fc86959a640 1 -- 172.21.15.134:0/3620164833 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc860001710 con 0x7fc86c0a98f0 2024-03-25T17:32:33.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.458+0000 7fc86959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:33.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.458+0000 7fc8708fb640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:33.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.458+0000 7fc8708fb640 4 set_mon_vals no callback set 2024-03-25T17:32:33.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.459+0000 7fc8708fb640 10 monclient: shutdown 2024-03-25T17:32:33.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.459+0000 7fc8708fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:33.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.459+0000 7fc8708fb640 1 -- 172.21.15.134:0/3620164833 >> v1:172.21.15.134:6789/0 conn(0x7fc86c0a98f0 legacy=0x7fc86c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:33.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.459+0000 7fc8708fb640 1 -- 172.21.15.134:0/3620164833 shutdown_connections 2024-03-25T17:32:33.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.459+0000 7fc8708fb640 1 -- 172.21.15.134:0/3620164833 >> 172.21.15.134:0/3620164833 conn(0x7fc86c0b7ea0 msgr2=0x7fc86c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:33.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.459+0000 7fc8708fb640 1 -- 172.21.15.134:0/3620164833 shutdown_connections 2024-03-25T17:32:33.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.459+0000 7fc8708fb640 1 -- 172.21.15.134:0/3620164833 wait complete. 2024-03-25T17:32:33.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 10 monclient: build_initial_monmap 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 10 monclient: monmap: 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 1 librados: starting msgr at 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 1 librados: starting objecter 2024-03-25T17:32:33.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 1 Processor -- start 2024-03-25T17:32:33.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 1 -- start start 2024-03-25T17:32:33.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 1 librados: setting wanted keys 2024-03-25T17:32:33.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 1 librados: calling monclient init 2024-03-25T17:32:33.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.460+0000 7fc8708fb640 10 monclient: init 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c139d20 con 0x7fc86c1430c0 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c13b840 con 0x7fc86c0a98f0 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c1303b0 con 0x7fc86c13f710 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:33.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.461+0000 7fc8708fb640 10 monclient(hunting): authenticate will time out at 5506.724112s 2024-03-25T17:32:33.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.468+0000 7fc86a59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc86c0a98f0 0x7fc86c12f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:45014/0 (socket says 172.21.15.134:45014) 2024-03-25T17:32:33.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.468+0000 7fc86a59c640 1 -- 172.21.15.134:0/1656306241 learned_addr learned my addr 172.21.15.134:0/1656306241 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:33.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.470+0000 7fc85affd640 10 client.?.objecter ms_handle_connect 0x7fc86c0a98f0 2024-03-25T17:32:33.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.470+0000 7fc85affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:33.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.485+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274129859 0 0) 0x7fc86c13b840 con 0x7fc86c0a98f0 2024-03-25T17:32:33.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.485+0000 7fc85affd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:33.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.485+0000 7fc85affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:33.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.485+0000 7fc85affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:33.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.485+0000 7fc85affd640 10 monclient(hunting): my global_id is 4366 2024-03-25T17:32:33.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.485+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84c004eb0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.490+0000 7fc85affd640 10 client.?.objecter ms_handle_connect 0x7fc86c13f710 2024-03-25T17:32:33.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.490+0000 7fc85affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:33.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.490+0000 7fc85affd640 10 client.?.objecter ms_handle_connect 0x7fc86c1430c0 2024-03-25T17:32:33.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.490+0000 7fc85affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:33.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.498+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402061694 0 0) 0x7fc86c1303b0 con 0x7fc86c13f710 2024-03-25T17:32:33.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.498+0000 7fc85affd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:33.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.498+0000 7fc85affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:33.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.498+0000 7fc85affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:33.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.498+0000 7fc85affd640 10 monclient(hunting): my global_id is 34337 2024-03-25T17:32:33.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.498+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c13b840 con 0x7fc86c13f710 2024-03-25T17:32:33.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.506+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500197117 0 0) 0x7fc84c004eb0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.506+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc86c1303b0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.506+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc860001d00 con 0x7fc86c0a98f0 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.506+0000 7fc85affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.506+0000 7fc85affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.506+0000 7fc85affd640 10 monclient(hunting): dump: 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:33.507 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:33.508 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:33.508 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:33.508 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.506+0000 7fc85affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.507+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387426434 0 0) 0x7fc86c139d20 con 0x7fc86c1430c0 2024-03-25T17:32:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.507+0000 7fc85affd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.507+0000 7fc85affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:33.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.507+0000 7fc85affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:33.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.507+0000 7fc85affd640 10 monclient(hunting): my global_id is 4347 2024-03-25T17:32:33.509 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.507+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84c004eb0 con 0x7fc86c1430c0 2024-03-25T17:32:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2116128117 0 0) 0x7fc86c1303b0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 >> v1:172.21.15.134:6790/0 conn(0x7fc86c13f710 legacy=0x7fc86c13bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 >> v1:172.21.15.134:6789/0 conn(0x7fc86c1430c0 legacy=0x7fc86c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 10 monclient: _finish_hunting 0 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 1 monclient: found mon.b 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0b71a0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 10 monclient: _finish_auth 0 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 10 monclient: _check_auth_tickets 2024-03-25T17:32:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc85affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 5 monclient: authenticate success, global_id 4366 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 10 monclient: _renew_subs 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c0b73f0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 10 monclient: _renew_subs 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c13c5a0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.512+0000 7fc8708fb640 1 librados: init done 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.513+0000 7fc8708fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.513+0000 7fc8708fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:33.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.513+0000 7fc8708fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:33.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.513+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c0627b0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.789+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8600038a0 con 0x7fc86c0a98f0 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.789+0000 7fc85affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.789+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc860003c70 con 0x7fc86c0a98f0 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.789+0000 7fc85affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.789+0000 7fc85affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.789+0000 7fc85affd640 10 monclient: dump: 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:33.790 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:33.791 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:33.791 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:33.791 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:33.791 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:33.791 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:33.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.789+0000 7fc868d99640 4 set_mon_vals no callback set 2024-03-25T17:32:33.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.790+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fc860003e90 con 0x7fc86c0a98f0 2024-03-25T17:32:33.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.790+0000 7fc869d9b640 10 monclient: get_auth_request con 0x7fc84c06de00 auth_method 0 2024-03-25T17:32:33.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.791+0000 7fc85affd640 10 client.4366.objecter ms_handle_connect 0x7fc84c06de00 2024-03-25T17:32:33.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.794+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fc860054900 con 0x7fc86c0a98f0 2024-03-25T17:32:33.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.794+0000 7fc85affd640 10 client.4366.objecter ms_dispatch 0x7fc86c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:33.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.794+0000 7fc85affd640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:33.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.794+0000 7fc85affd640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:33.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.795+0000 7fc85affd640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T17:32:33.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.795+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc860050410 con 0x7fc86c0a98f0 2024-03-25T17:32:33.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.795+0000 7fc85affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:33.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:33.795+0000 7fc85affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:34.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.040+0000 7fc8708fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:34.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.040+0000 7fc8708fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:34.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.040+0000 7fc8708fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:34.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.040+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc86c13de20 con 0x7fc86c0a98f0 2024-03-25T17:32:34.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.070+0000 7fc85affd640 1 -- 172.21.15.134:0/1656306241 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fc860052340 con 0x7fc86c0a98f0 2024-03-25T17:32:34.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.070+0000 7fc85affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:34.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.070+0000 7fc85affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:34.107 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:34.107 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:34.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 10 librados: watch_flush enter 2024-03-25T17:32:34.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 10 librados: watch_flush exit 2024-03-25T17:32:34.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T17:32:34.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 10 client.4366.objecter successfully canceled tick 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 >> v1:172.21.15.134:6800/610837766 conn(0x7fc84c06de00 legacy=0x7fc84c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 10 monclient: shutdown 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 >> v1:172.21.15.159:6789/0 conn(0x7fc86c0a98f0 legacy=0x7fc86c12f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 shutdown_connections 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 >> 172.21.15.134:0/1656306241 conn(0x7fc86c0b7ea0 msgr2=0x7fc86c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 shutdown_connections 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.072+0000 7fc8708fb640 1 -- 172.21.15.134:0/1656306241 wait complete. 2024-03-25T17:32:34.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:34.073+0000 7fc8708fb640 1 librados: shutdown 2024-03-25T17:32:34.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:40.916 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:41.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.030+0000 7f740d606640 10 monclient: get_monmap_and_config 2024-03-25T17:32:41.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.031+0000 7f740d606640 10 monclient: build_initial_monmap 2024-03-25T17:32:41.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.031+0000 7f740d606640 10 monclient: monmap: 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:41.032018+0000 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:41.032018+0000 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:41.032 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:41.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.031+0000 7f740d606640 1 Processor -- start 2024-03-25T17:32:41.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.031+0000 7f740d606640 1 -- start start 2024-03-25T17:32:41.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.031+0000 7f740d606640 10 monclient: init 2024-03-25T17:32:41.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:41.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:32:41.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 10 monclient(hunting): picked mon.noname-b con 0x7f74080a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:41.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 10 monclient(hunting): picked mon.noname-a con 0x7f740813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:41.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 10 monclient(hunting): picked mon.noname-c con 0x7f74081430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:41.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7408142520 con 0x7f740813f710 2024-03-25T17:32:41.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080b28a0 con 0x7f74081430c0 2024-03-25T17:32:41.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74080b2a30 con 0x7f74080a98f0 2024-03-25T17:32:41.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:41.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.032+0000 7f740d606640 10 monclient(hunting): authenticate will time out at 5514.295603s 2024-03-25T17:32:41.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.050+0000 7f74067fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f740813f710 0x7f74081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:51150/0 (socket says 172.21.15.134:51150) 2024-03-25T17:32:41.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.051+0000 7f74067fc640 1 -- 172.21.15.134:0/2284404715 learned_addr learned my addr 172.21.15.134:0/2284404715 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:41.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.056+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893290899 0 0) 0x7f7408142520 con 0x7f740813f710 2024-03-25T17:32:41.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.056+0000 7f7405ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:32:41.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.056+0000 7f7405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:41.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.056+0000 7f7405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:41.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.056+0000 7f7405ffb640 10 monclient(hunting): my global_id is 4350 2024-03-25T17:32:41.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.056+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f0003520 con 0x7f740813f710 2024-03-25T17:32:41.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.069+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 979503995 0 0) 0x7f74080b28a0 con 0x7f74081430c0 2024-03-25T17:32:41.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.069+0000 7f7405ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:32:41.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.069+0000 7f7405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:41.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.069+0000 7f7405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:41.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.069+0000 7f7405ffb640 10 monclient(hunting): my global_id is 4369 2024-03-25T17:32:41.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.069+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7408142520 con 0x7f74081430c0 2024-03-25T17:32:41.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.070+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465785320 0 0) 0x7f74080b2a30 con 0x7f74080a98f0 2024-03-25T17:32:41.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.070+0000 7f7405ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:32:41.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.070+0000 7f7405ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:41.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.070+0000 7f7405ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:41.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.070+0000 7f7405ffb640 10 monclient(hunting): my global_id is 34340 2024-03-25T17:32:41.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.070+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74080b28a0 con 0x7f74080a98f0 2024-03-25T17:32:41.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3805958111 0 0) 0x7f73f0003520 con 0x7f740813f710 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 >> v1:172.21.15.134:6790/0 conn(0x7f74080a98f0 legacy=0x7f74080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 >> v1:172.21.15.159:6789/0 conn(0x7f74081430c0 legacy=0x7f740814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 1 monclient: found mon.noname-a 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74080b44c0 con 0x7f740813f710 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 10 monclient: _finish_auth 0 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f73fc001690 con 0x7f740813f710 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f7405ffb640 10 monclient: dump: 2024-03-25T17:32:41.076 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f740d606640 5 monclient: authenticate success, global_id 4350 2024-03-25T17:32:41.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.075+0000 7f740d606640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73fc0017f0 con 0x7f740813f710 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f7405ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f7405ffb640 1 -- 172.21.15.134:0/2284404715 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f73fc001c00 con 0x7f740813f710 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f7405ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f7405ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f7405ffb640 10 monclient: dump: 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:41.093 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:41.094 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:41.094 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:41.094 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:41.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f740d606640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:41.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.092+0000 7f740d606640 4 set_mon_vals no callback set 2024-03-25T17:32:41.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.093+0000 7f740d606640 10 monclient: shutdown 2024-03-25T17:32:41.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.093+0000 7f740d606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:41.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.093+0000 7f740d606640 1 -- 172.21.15.134:0/2284404715 >> v1:172.21.15.134:6789/0 conn(0x7f740813f710 legacy=0x7f74081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.093+0000 7f740d606640 1 -- 172.21.15.134:0/2284404715 shutdown_connections 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.093+0000 7f740d606640 1 -- 172.21.15.134:0/2284404715 >> 172.21.15.134:0/2284404715 conn(0x7f74080b7ea0 msgr2=0x7f74080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.093+0000 7f740d606640 1 -- 172.21.15.134:0/2284404715 shutdown_connections 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.093+0000 7f740d606640 1 -- 172.21.15.134:0/2284404715 wait complete. 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.094+0000 7f740d606640 10 monclient: build_initial_monmap 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.094+0000 7f740d606640 10 monclient: monmap: 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:41.096 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.094+0000 7f740d606640 1 librados: starting msgr at 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.094+0000 7f740d606640 1 librados: starting objecter 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 1 Processor -- start 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 1 -- start start 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 1 librados: setting wanted keys 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 1 librados: calling monclient init 2024-03-25T17:32:41.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 10 monclient: init 2024-03-25T17:32:41.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:41.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:32:41.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.095+0000 7f740d606640 10 monclient(hunting): picked mon.noname-a con 0x7f74080a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:41.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.096+0000 7f740d606640 10 monclient(hunting): picked mon.noname-b con 0x7f740813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:41.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.096+0000 7f740d606640 10 monclient(hunting): picked mon.noname-c con 0x7f74081430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:41.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.096+0000 7f740d606640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740806f010 con 0x7f74080a98f0 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.096+0000 7f740d606640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740806fc50 con 0x7f740813f710 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.096+0000 7f740d606640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f740806ff30 con 0x7f74081430c0 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.096+0000 7f740d606640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.096+0000 7f740d606640 10 monclient(hunting): authenticate will time out at 5514.358888s 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.098+0000 7f74067fc640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f740813f710 0x7f740814ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:56112/0 (socket says 172.21.15.134:56112) 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.098+0000 7f74067fc640 1 -- 172.21.15.134:0/360339435 learned_addr learned my addr 172.21.15.134:0/360339435 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.099+0000 7f73ef7fe640 10 client.?.objecter ms_handle_connect 0x7f740813f710 2024-03-25T17:32:41.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.099+0000 7f73ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:41.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.118+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267446324 0 0) 0x7f740806fc50 con 0x7f740813f710 2024-03-25T17:32:41.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.118+0000 7f73ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:32:41.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.118+0000 7f73ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:41.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.118+0000 7f73ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:41.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.118+0000 7f73ef7fe640 10 monclient(hunting): my global_id is 4372 2024-03-25T17:32:41.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.119+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f4004eb0 con 0x7f740813f710 2024-03-25T17:32:41.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.122+0000 7f73ef7fe640 10 client.?.objecter ms_handle_connect 0x7f74081430c0 2024-03-25T17:32:41.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.122+0000 7f73ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:41.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.126+0000 7f73ef7fe640 10 client.?.objecter ms_handle_connect 0x7f74080a98f0 2024-03-25T17:32:41.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.126+0000 7f73ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:41.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.133+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3103350108 0 0) 0x7f73f4004eb0 con 0x7f740813f710 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.133+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f740806fc50 con 0x7f740813f710 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.133+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f73fc001bc0 con 0x7f740813f710 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.133+0000 7f73ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.133+0000 7f73ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.133+0000 7f73ef7fe640 10 monclient(hunting): dump: 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:41.134 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:41.135 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:41.135 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:41.135 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:41.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.133+0000 7f73ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:41.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.138+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1868749656 0 0) 0x7f740806ff30 con 0x7f74081430c0 2024-03-25T17:32:41.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.138+0000 7f73ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:32:41.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.138+0000 7f73ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:41.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.138+0000 7f73ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:41.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.138+0000 7f73ef7fe640 10 monclient(hunting): my global_id is 34343 2024-03-25T17:32:41.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.138+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f4004eb0 con 0x7f74081430c0 2024-03-25T17:32:41.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.154+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522887880 0 0) 0x7f740806f010 con 0x7f74080a98f0 2024-03-25T17:32:41.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.154+0000 7f73ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:32:41.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.154+0000 7f73ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:41.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.154+0000 7f73ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:41.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.154+0000 7f73ef7fe640 10 monclient(hunting): my global_id is 4353 2024-03-25T17:32:41.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.154+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f740806ff30 con 0x7f74080a98f0 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1326431941 0 0) 0x7f740806fc50 con 0x7f740813f710 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 >> v1:172.21.15.134:6790/0 conn(0x7f74081430c0 legacy=0x7f74081df0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 >> v1:172.21.15.134:6789/0 conn(0x7f74080a98f0 legacy=0x7f74080706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 1 monclient: found mon.b 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7408070db0 con 0x7f740813f710 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 10 monclient: _finish_auth 0 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f73ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:41.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 5 monclient: authenticate success, global_id 4372 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 10 monclient: _renew_subs 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f740814e710 con 0x7f740813f710 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 10 monclient: _renew_subs 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f740806e540 con 0x7f740813f710 2024-03-25T17:32:41.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.162+0000 7f740d606640 1 librados: init done 2024-03-25T17:32:41.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.163+0000 7f740d606640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:41.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.163+0000 7f740d606640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:41.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.163+0000 7f740d606640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:41.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.163+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f740806e010 con 0x7f740813f710 2024-03-25T17:32:41.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.452+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73fc0011f0 con 0x7f740813f710 2024-03-25T17:32:41.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.452+0000 7f73ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:41.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.452+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f73fc004a90 con 0x7f740813f710 2024-03-25T17:32:41.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.452+0000 7f73ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:41.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.452+0000 7f73ef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.452+0000 7f73ef7fe640 10 monclient: dump: 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:41.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.452+0000 7f74057fa640 4 set_mon_vals no callback set 2024-03-25T17:32:41.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.453+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f73fc004cb0 con 0x7f740813f710 2024-03-25T17:32:41.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.454+0000 7f7406ffd640 10 monclient: get_auth_request con 0x7f73f406ddc0 auth_method 0 2024-03-25T17:32:41.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.454+0000 7f73ef7fe640 10 client.4372.objecter ms_handle_connect 0x7f73f406ddc0 2024-03-25T17:32:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.460+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f73fc054840 con 0x7f740813f710 2024-03-25T17:32:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.460+0000 7f73ef7fe640 10 client.4372.objecter ms_dispatch 0x7f7408000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.460+0000 7f73ef7fe640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.460+0000 7f73ef7fe640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.460+0000 7f73ef7fe640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T17:32:41.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.461+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73fc050180 con 0x7f740813f710 2024-03-25T17:32:41.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.461+0000 7f73ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:41.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.461+0000 7f73ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:41.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.680+0000 7f740d606640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:41.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.680+0000 7f740d606640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:41.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.680+0000 7f740d606640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:41.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.680+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f740814e8a0 con 0x7f740813f710 2024-03-25T17:32:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.710+0000 7f73ef7fe640 1 -- 172.21.15.134:0/360339435 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f73fc052280 con 0x7f740813f710 2024-03-25T17:32:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.710+0000 7f73ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:41.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.710+0000 7f73ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:41.711 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:41.711 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:41.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.712+0000 7f740d606640 10 librados: watch_flush enter 2024-03-25T17:32:41.713 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.712+0000 7f740d606640 10 librados: watch_flush exit 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.712+0000 7f740d606640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.712+0000 7f740d606640 10 client.4372.objecter successfully canceled tick 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 >> v1:172.21.15.134:6800/610837766 conn(0x7f73f406ddc0 legacy=0x7f73f4070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 10 monclient: shutdown 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 >> v1:172.21.15.159:6789/0 conn(0x7f740813f710 legacy=0x7f740814ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 shutdown_connections 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 >> 172.21.15.134:0/360339435 conn(0x7f74080b7ea0 msgr2=0x7f740813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 shutdown_connections 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 1 -- 172.21.15.134:0/360339435 wait complete. 2024-03-25T17:32:41.714 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:41.713+0000 7f740d606640 1 librados: shutdown 2024-03-25T17:32:41.726 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:44.535 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:49.551 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.540 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:49.557 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.579 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:03:49.596 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.600 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:49.629 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.604 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:49.621 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.609 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:49.638 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.611 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:49.628 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.634 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:03:49.651 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:44.647 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:03:49.675 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:32:48.531 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.644+0000 7f6950e5c640 10 monclient: get_monmap_and_config 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.644+0000 7f6950e5c640 10 monclient: build_initial_monmap 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.644+0000 7f6950e5c640 10 monclient: monmap: 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:48.645692+0000 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:48.645692+0000 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:48.645 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:48.646 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:48.646 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:48.646 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.645+0000 7f6950e5c640 1 Processor -- start 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.645+0000 7f6950e5c640 1 -- start start 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.645+0000 7f6950e5c640 10 monclient: init 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f694c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f694c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f694c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:48.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c142520 con 0x7f694c13f710 2024-03-25T17:32:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c0b28a0 con 0x7f694c0a98f0 2024-03-25T17:32:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c0b2a30 con 0x7f694c1430c0 2024-03-25T17:32:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:48.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.646+0000 7f6950e5c640 10 monclient(hunting): authenticate will time out at 5521.909089s 2024-03-25T17:32:48.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.664+0000 7f694a59c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f694c0a98f0 0x7f694c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:60762/0 (socket says 172.21.15.134:60762) 2024-03-25T17:32:48.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.664+0000 7f694a59c640 1 -- 172.21.15.134:0/3691205190 learned_addr learned my addr 172.21.15.134:0/3691205190 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:48.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.678+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2513093961 0 0) 0x7f694c0b2a30 con 0x7f694c1430c0 2024-03-25T17:32:48.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.678+0000 7f6949d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:32:48.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.678+0000 7f6949d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:48.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.678+0000 7f6949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:48.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.678+0000 7f6949d9b640 10 monclient(hunting): my global_id is 34346 2024-03-25T17:32:48.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.678+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6938003520 con 0x7f694c1430c0 2024-03-25T17:32:48.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.681+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 57539109 0 0) 0x7f694c0b28a0 con 0x7f694c0a98f0 2024-03-25T17:32:48.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.681+0000 7f6949d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:32:48.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.681+0000 7f6949d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:48.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.681+0000 7f6949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:48.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.681+0000 7f6949d9b640 10 monclient(hunting): my global_id is 4375 2024-03-25T17:32:48.682 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.681+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f694c0b2a30 con 0x7f694c0a98f0 2024-03-25T17:32:48.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1572542526 0 0) 0x7f694c0b2a30 con 0x7f694c0a98f0 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 >> v1:172.21.15.134:6790/0 conn(0x7f694c1430c0 legacy=0x7f694c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 >> v1:172.21.15.134:6789/0 conn(0x7f694c13f710 legacy=0x7f694c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 1 monclient: found mon.noname-c 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c0b44c0 con 0x7f694c0a98f0 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 10 monclient: _finish_auth 0 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f693c001670 con 0x7f694c0a98f0 2024-03-25T17:32:48.688 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6950e5c640 5 monclient: authenticate success, global_id 4375 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6950e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6949d9b640 10 monclient: dump: 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:48.689 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:48.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.687+0000 7f6950e5c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:48.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f693c0017d0 con 0x7f694c0a98f0 2024-03-25T17:32:48.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6949d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:48.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6949d9b640 1 -- 172.21.15.134:0/3691205190 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f693c001ba0 con 0x7f694c0a98f0 2024-03-25T17:32:48.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6949d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6949d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6949d9b640 10 monclient: dump: 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:48.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6950e5c640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:48.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.705+0000 7f6950e5c640 4 set_mon_vals no callback set 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.706+0000 7f6950e5c640 10 monclient: shutdown 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.706+0000 7f6950e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.706+0000 7f6950e5c640 1 -- 172.21.15.134:0/3691205190 >> v1:172.21.15.159:6789/0 conn(0x7f694c0a98f0 legacy=0x7f694c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.706+0000 7f6950e5c640 1 -- 172.21.15.134:0/3691205190 shutdown_connections 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.706+0000 7f6950e5c640 1 -- 172.21.15.134:0/3691205190 >> 172.21.15.134:0/3691205190 conn(0x7f694c0b7ea0 msgr2=0x7f694c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.706+0000 7f6950e5c640 1 -- 172.21.15.134:0/3691205190 shutdown_connections 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.706+0000 7f6950e5c640 1 -- 172.21.15.134:0/3691205190 wait complete. 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 10 monclient: build_initial_monmap 2024-03-25T17:32:48.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 10 monclient: monmap: 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 1 librados: starting msgr at 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 1 librados: starting objecter 2024-03-25T17:32:48.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 1 Processor -- start 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 1 -- start start 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 1 librados: setting wanted keys 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 1 librados: calling monclient init 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.707+0000 7f6950e5c640 10 monclient: init 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 10 monclient(hunting): picked mon.noname-b con 0x7f694c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 10 monclient(hunting): picked mon.noname-a con 0x7f694c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 10 monclient(hunting): picked mon.noname-c con 0x7f694c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c1390e0 con 0x7f694c13f710 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c1349e0 con 0x7f694c0a98f0 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694c139350 con 0x7f694c1430c0 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:48.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.708+0000 7f6950e5c640 10 monclient(hunting): authenticate will time out at 5521.971140s 2024-03-25T17:32:48.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.712+0000 7f694ad9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f694c1430c0 0x7f694c1342d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:47948/0 (socket says 172.21.15.134:47948) 2024-03-25T17:32:48.712 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.712+0000 7f694ad9d640 1 -- 172.21.15.134:0/3363539863 learned_addr learned my addr 172.21.15.134:0/3363539863 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:48.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.718+0000 7f6942ffd640 10 client.?.objecter ms_handle_connect 0x7f694c1430c0 2024-03-25T17:32:48.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.718+0000 7f6942ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:48.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.728+0000 7f6942ffd640 10 client.?.objecter ms_handle_connect 0x7f694c13f710 2024-03-25T17:32:48.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.728+0000 7f6942ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:48.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.729+0000 7f6942ffd640 10 client.?.objecter ms_handle_connect 0x7f694c0a98f0 2024-03-25T17:32:48.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.729+0000 7f6942ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:48.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.735+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774562658 0 0) 0x7f694c139350 con 0x7f694c1430c0 2024-03-25T17:32:48.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.735+0000 7f6942ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:48.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.735+0000 7f6942ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:48.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.735+0000 7f6942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:48.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.735+0000 7f6942ffd640 10 monclient(hunting): my global_id is 34349 2024-03-25T17:32:48.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.735+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f692c004eb0 con 0x7f694c1430c0 2024-03-25T17:32:48.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.737+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741861881 0 0) 0x7f694c1390e0 con 0x7f694c13f710 2024-03-25T17:32:48.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.737+0000 7f6942ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:48.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.737+0000 7f6942ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:48.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.737+0000 7f6942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:48.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.737+0000 7f6942ffd640 10 monclient(hunting): my global_id is 4359 2024-03-25T17:32:48.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.737+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f694c139350 con 0x7f694c13f710 2024-03-25T17:32:48.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.740+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769065911 0 0) 0x7f694c1349e0 con 0x7f694c0a98f0 2024-03-25T17:32:48.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.740+0000 7f6942ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:32:48.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.740+0000 7f6942ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:48.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.740+0000 7f6942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:48.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.740+0000 7f6942ffd640 10 monclient(hunting): my global_id is 4378 2024-03-25T17:32:48.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.740+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f694c1390e0 con 0x7f694c0a98f0 2024-03-25T17:32:48.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2314206275 0 0) 0x7f694c139350 con 0x7f694c13f710 2024-03-25T17:32:48.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f694c1349e0 con 0x7f694c13f710 2024-03-25T17:32:48.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f69340018a0 con 0x7f694c13f710 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 10 monclient(hunting): dump: 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:48.757 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848604976 0 0) 0x7f694c1390e0 con 0x7f694c0a98f0 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f694c139350 con 0x7f694c0a98f0 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f693c001ae0 con 0x7f694c0a98f0 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.755+0000 7f6942ffd640 10 monclient(hunting): dump: 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:48.758 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:48.759 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:48.759 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:48.759 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.765+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084398339 0 0) 0x7f692c004eb0 con 0x7f694c1430c0 2024-03-25T17:32:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.765+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f694c1390e0 con 0x7f694c1430c0 2024-03-25T17:32:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.765+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6944001c70 con 0x7f694c1430c0 2024-03-25T17:32:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.765+0000 7f6942ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.765+0000 7f6942ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:32:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.765+0000 7f6942ffd640 10 monclient(hunting): dump: 2024-03-25T17:32:48.766 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:48.767 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:48.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1833741033 0 0) 0x7f694c139350 con 0x7f694c0a98f0 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 >> v1:172.21.15.134:6790/0 conn(0x7f694c1430c0 legacy=0x7f694c1342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 >> v1:172.21.15.134:6789/0 conn(0x7f694c13f710 legacy=0x7f694c133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 1 monclient: found mon.b 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c139630 con 0x7f694c0a98f0 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 10 monclient: _finish_auth 0 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6942ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 5 monclient: authenticate success, global_id 4378 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 10 monclient: _renew_subs 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:48.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f694c139910 con 0x7f694c0a98f0 2024-03-25T17:32:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 10 monclient: _renew_subs 2024-03-25T17:32:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f694c137800 con 0x7f694c0a98f0 2024-03-25T17:32:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.767+0000 7f6950e5c640 1 librados: init done 2024-03-25T17:32:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.768+0000 7f6950e5c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.768+0000 7f6950e5c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:48.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.768+0000 7f6950e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:48.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:48.768+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694c074cc0 con 0x7f694c0a98f0 2024-03-25T17:32:49.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.042+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f693c001c40 con 0x7f694c0a98f0 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.042+0000 7f6942ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.042+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f693c004c30 con 0x7f694c0a98f0 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.042+0000 7f6942ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.042+0000 7f6942ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.042+0000 7f6942ffd640 10 monclient: dump: 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:49.043 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:49.044 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:49.044 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:49.044 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:49.044 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:49.044 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:49.044 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:49.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.042+0000 7f694959a640 4 set_mon_vals no callback set 2024-03-25T17:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.043+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f693c01ac00 con 0x7f694c0a98f0 2024-03-25T17:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.043+0000 7f6943fff640 10 monclient: get_auth_request con 0x7f692c06e160 auth_method 0 2024-03-25T17:32:49.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.044+0000 7f6942ffd640 10 client.4378.objecter ms_handle_connect 0x7f692c06e160 2024-03-25T17:32:49.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.050+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f693c054780 con 0x7f694c0a98f0 2024-03-25T17:32:49.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.050+0000 7f6942ffd640 10 client.4378.objecter ms_dispatch 0x7f694c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:49.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.050+0000 7f6942ffd640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:49.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.050+0000 7f6942ffd640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:49.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.050+0000 7f6942ffd640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T17:32:49.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.051+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f693c0500c0 con 0x7f694c0a98f0 2024-03-25T17:32:49.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.051+0000 7f6942ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:49.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.051+0000 7f6942ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:49.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.261+0000 7f6950e5c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:49.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.261+0000 7f6950e5c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:49.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.261+0000 7f6950e5c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:32:49.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.261+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f694c138910 con 0x7f694c0a98f0 2024-03-25T17:32:49.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.369+0000 7f6942ffd640 1 -- 172.21.15.134:0/3363539863 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f693c0521c0 con 0x7f694c0a98f0 2024-03-25T17:32:49.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.369+0000 7f6942ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:49.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.369+0000 7f6942ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:49.370 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:49.370 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:49.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 10 librados: watch_flush enter 2024-03-25T17:32:49.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 10 librados: watch_flush exit 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 10 client.4378.objecter successfully canceled tick 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 >> v1:172.21.15.134:6800/610837766 conn(0x7f692c06e160 legacy=0x7f692c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 10 monclient: shutdown 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 >> v1:172.21.15.159:6789/0 conn(0x7f694c0a98f0 legacy=0x7f694c1d72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 shutdown_connections 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 >> 172.21.15.134:0/3363539863 conn(0x7f694c0b7ea0 msgr2=0x7f694c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.371+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 shutdown_connections 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.372+0000 7f6950e5c640 1 -- 172.21.15.134:0/3363539863 wait complete. 2024-03-25T17:32:49.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:49.372+0000 7f6950e5c640 1 librados: shutdown 2024-03-25T17:32:49.386 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:32:56.193 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.312+0000 7f8a85c96640 10 monclient: get_monmap_and_config 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.312+0000 7f8a85c96640 10 monclient: build_initial_monmap 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.312+0000 7f8a85c96640 10 monclient: monmap: 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:32:56.313678+0000 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:32:56.313678+0000 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:56.313 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:56.314 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:56.314 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:32:56.314 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:32:56.314 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:56.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.313+0000 7f8a85c96640 1 Processor -- start 2024-03-25T17:32:56.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.313+0000 7f8a85c96640 1 -- start start 2024-03-25T17:32:56.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.313+0000 7f8a85c96640 10 monclient: init 2024-03-25T17:32:56.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:56.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:32:56.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 10 monclient(hunting): picked mon.noname-c con 0x7f8a800a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:56.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 10 monclient(hunting): picked mon.noname-b con 0x7f8a8013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:56.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 10 monclient(hunting): picked mon.noname-a con 0x7f8a801430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:56.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a80142520 con 0x7f8a801430c0 2024-03-25T17:32:56.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a800b28a0 con 0x7f8a800a98f0 2024-03-25T17:32:56.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a800b2a30 con 0x7f8a8013f710 2024-03-25T17:32:56.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:56.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.314+0000 7f8a85c96640 10 monclient(hunting): authenticate will time out at 5529.577170s 2024-03-25T17:32:56.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.324+0000 7f8a7ffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8a801430c0 0x7f8a8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:53984/0 (socket says 172.21.15.134:53984) 2024-03-25T17:32:56.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.324+0000 7f8a7ffff640 1 -- 172.21.15.134:0/646616386 learned_addr learned my addr 172.21.15.134:0/646616386 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:56.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.334+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3275545490 0 0) 0x7f8a80142520 con 0x7f8a801430c0 2024-03-25T17:32:56.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.334+0000 7f8a7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:32:56.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.334+0000 7f8a7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:56.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.334+0000 7f8a7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:56.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.334+0000 7f8a7e7fc640 10 monclient(hunting): my global_id is 4362 2024-03-25T17:32:56.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.334+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a68003520 con 0x7f8a801430c0 2024-03-25T17:32:56.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1453988966 0 0) 0x7f8a68003520 con 0x7f8a801430c0 2024-03-25T17:32:56.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 >> v1:172.21.15.134:6790/0 conn(0x7f8a8013f710 legacy=0x7f8a801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:32:56.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 >> v1:172.21.15.159:6789/0 conn(0x7f8a800a98f0 legacy=0x7f8a800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:56.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:32:56.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 1 monclient: found mon.noname-a 2024-03-25T17:32:56.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:56.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800b44c0 con 0x7f8a801430c0 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8a78001a10 con 0x7f8a801430c0 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a7e7fc640 10 monclient: dump: 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:56.342 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:56.343 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:56.343 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:56.343 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:56.343 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:56.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a85c96640 5 monclient: authenticate success, global_id 4362 2024-03-25T17:32:56.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.340+0000 7f8a85c96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a78001be0 con 0x7f8a801430c0 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a7e7fc640 1 -- 172.21.15.134:0/646616386 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8a78002030 con 0x7f8a801430c0 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a7e7fc640 10 monclient: dump: 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:56.363 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a85c96640 10 monclient: get_monmap_and_config success 2024-03-25T17:32:56.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.362+0000 7f8a85c96640 4 set_mon_vals no callback set 2024-03-25T17:32:56.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.363+0000 7f8a85c96640 10 monclient: shutdown 2024-03-25T17:32:56.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.363+0000 7f8a85c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.363+0000 7f8a85c96640 1 -- 172.21.15.134:0/646616386 >> v1:172.21.15.134:6789/0 conn(0x7f8a801430c0 legacy=0x7f8a8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.363+0000 7f8a85c96640 1 -- 172.21.15.134:0/646616386 shutdown_connections 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.363+0000 7f8a85c96640 1 -- 172.21.15.134:0/646616386 >> 172.21.15.134:0/646616386 conn(0x7f8a800b7ea0 msgr2=0x7f8a800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.363+0000 7f8a85c96640 1 -- 172.21.15.134:0/646616386 shutdown_connections 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.363+0000 7f8a85c96640 1 -- 172.21.15.134:0/646616386 wait complete. 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.364+0000 7f8a85c96640 10 monclient: build_initial_monmap 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.364+0000 7f8a85c96640 10 monclient: monmap: 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:32:56.366 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.364+0000 7f8a85c96640 1 librados: starting msgr at 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.364+0000 7f8a85c96640 1 librados: starting objecter 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.364+0000 7f8a85c96640 1 Processor -- start 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 1 -- start start 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 1 librados: setting wanted keys 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 1 librados: calling monclient init 2024-03-25T17:32:56.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient: init 2024-03-25T17:32:56.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient: _reopen_session rank -1 2024-03-25T17:32:56.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:32:56.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient(hunting): picked mon.noname-a con 0x7f8a800a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:32:56.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient(hunting): picked mon.noname-c con 0x7f8a8013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:32:56.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient(hunting): picked mon.noname-b con 0x7f8a801430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:32:56.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a801db660 con 0x7f8a800a98f0 2024-03-25T17:32:56.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a801db7f0 con 0x7f8a801430c0 2024-03-25T17:32:56.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a80138c40 con 0x7f8a8013f710 2024-03-25T17:32:56.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient(hunting): _renew_subs 2024-03-25T17:32:56.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.365+0000 7f8a85c96640 10 monclient(hunting): authenticate will time out at 5529.628684s 2024-03-25T17:32:56.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.378+0000 7f8a7effd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f8a8013f710 0x7f8a80138530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:51736/0 (socket says 172.21.15.134:51736) 2024-03-25T17:32:56.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.378+0000 7f8a7effd640 1 -- 172.21.15.134:0/362725726 learned_addr learned my addr 172.21.15.134:0/362725726 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:32:56.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.379+0000 7f8a67fff640 10 client.?.objecter ms_handle_connect 0x7f8a8013f710 2024-03-25T17:32:56.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.379+0000 7f8a67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:56.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.381+0000 7f8a67fff640 10 client.?.objecter ms_handle_connect 0x7f8a800a98f0 2024-03-25T17:32:56.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.381+0000 7f8a67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:56.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 client.?.objecter ms_handle_connect 0x7f8a801430c0 2024-03-25T17:32:56.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806786095 0 0) 0x7f8a801db660 con 0x7f8a800a98f0 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): my global_id is 4365 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a6c004eb0 con 0x7f8a800a98f0 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733799347 0 0) 0x7f8a80138c40 con 0x7f8a8013f710 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:56.390 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:56.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:56.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 10 monclient(hunting): my global_id is 34352 2024-03-25T17:32:56.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.389+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a801db660 con 0x7f8a8013f710 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.397+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2942780635 0 0) 0x7f8a801db660 con 0x7f8a8013f710 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.397+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a80138c40 con 0x7f8a8013f710 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.397+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8a740018e0 con 0x7f8a8013f710 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.397+0000 7f8a67fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.397+0000 7f8a67fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.397+0000 7f8a67fff640 10 monclient(hunting): dump: 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:56.398 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:56.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.397+0000 7f8a67fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:32:56.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.409+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051641594 0 0) 0x7f8a801db7f0 con 0x7f8a801430c0 2024-03-25T17:32:56.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.409+0000 7f8a67fff640 10 monclient(hunting): _negotiate 2024-03-25T17:32:56.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.409+0000 7f8a67fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:32:56.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.409+0000 7f8a67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:32:56.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.409+0000 7f8a67fff640 10 monclient(hunting): my global_id is 4384 2024-03-25T17:32:56.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.409+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a801db660 con 0x7f8a801430c0 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.421+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1067614469 0 0) 0x7f8a80138c40 con 0x7f8a8013f710 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 >> v1:172.21.15.159:6789/0 conn(0x7f8a801430c0 legacy=0x7f8a801e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 >> v1:172.21.15.134:6789/0 conn(0x7f8a800a98f0 legacy=0x7f8a801daf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 10 monclient: _finish_hunting 0 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 1 monclient: found mon.c 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a80138dd0 con 0x7f8a8013f710 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 10 monclient: _finish_auth 0 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 10 monclient: _check_auth_tickets 2024-03-25T17:32:56.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 5 monclient: authenticate success, global_id 34352 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 10 monclient: _renew_subs 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a80137e70 con 0x7f8a8013f710 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 10 client.34352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 10 monclient: _renew_subs 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a801e3520 con 0x7f8a8013f710 2024-03-25T17:32:56.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.422+0000 7f8a85c96640 1 librados: init done 2024-03-25T17:32:56.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.423+0000 7f8a85c96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:56.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.423+0000 7f8a85c96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:56.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.423+0000 7f8a85c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:32:56.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.423+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a800b3ef0 con 0x7f8a8013f710 2024-03-25T17:32:56.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.728+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a740035f0 con 0x7f8a8013f710 2024-03-25T17:32:56.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.728+0000 7f8a67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:32:56.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.728+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8a74003940 con 0x7f8a8013f710 2024-03-25T17:32:56.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.728+0000 7f8a67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:32:56.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.728+0000 7f8a67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:32:56.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.728+0000 7f8a67fff640 10 monclient: dump: 2024-03-25T17:32:56.729 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:32:56.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.728+0000 7f8a7dffb640 4 set_mon_vals no callback set 2024-03-25T17:32:56.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.729+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f8a74003b60 con 0x7f8a8013f710 2024-03-25T17:32:56.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.730+0000 7f8a7f7fe640 10 monclient: get_auth_request con 0x7f8a6c06de10 auth_method 0 2024-03-25T17:32:56.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.730+0000 7f8a67fff640 10 client.34352.objecter ms_handle_connect 0x7f8a6c06de10 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f8a74054720 con 0x7f8a8013f710 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 10 client.34352.objecter ms_dispatch 0x7f8a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 3 client.34352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 3 client.34352.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 20 client.34352.objecter dump_active .. 0 homeless 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a74050230 con 0x7f8a8013f710 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:32:56.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.737+0000 7f8a67fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:32:56.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.963+0000 7f8a85c96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:56.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.963+0000 7f8a85c96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:56.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.963+0000 7f8a85c96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:32:56.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.963+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a80138000 con 0x7f8a8013f710 2024-03-25T17:32:56.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.986+0000 7f8a67fff640 1 -- 172.21.15.134:0/362725726 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f8a74052160 con 0x7f8a8013f710 2024-03-25T17:32:56.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.986+0000 7f8a67fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:32:56.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.986+0000 7f8a67fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:32:56.987 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:32:56.987 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:32:56.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.988+0000 7f8a85c96640 10 librados: watch_flush enter 2024-03-25T17:32:56.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.988+0000 7f8a85c96640 10 librados: watch_flush exit 2024-03-25T17:32:56.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.988+0000 7f8a85c96640 20 client.34352.objecter shutdown clearing up homeless session... 2024-03-25T17:32:56.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.988+0000 7f8a85c96640 10 client.34352.objecter successfully canceled tick 2024-03-25T17:32:56.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 >> v1:172.21.15.134:6800/610837766 conn(0x7f8a6c06de10 legacy=0x7f8a6c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:56.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 10 monclient: shutdown 2024-03-25T17:32:56.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:32:56.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 >> v1:172.21.15.134:6790/0 conn(0x7f8a8013f710 legacy=0x7f8a80138530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:32:56.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 shutdown_connections 2024-03-25T17:32:56.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 >> 172.21.15.134:0/362725726 conn(0x7f8a800b7ea0 msgr2=0x7f8a80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:32:56.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 shutdown_connections 2024-03-25T17:32:56.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 1 -- 172.21.15.134:0/362725726 wait complete. 2024-03-25T17:32:56.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:32:56.989+0000 7f8a85c96640 1 librados: shutdown 2024-03-25T17:32:57.002 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:03.808 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.925+0000 7f544058f640 10 monclient: get_monmap_and_config 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.925+0000 7f544058f640 10 monclient: build_initial_monmap 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.925+0000 7f544058f640 10 monclient: monmap: 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:03.926580+0000 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:03.926580+0000 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:03.926 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:03.927 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:03.927 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:03.927 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:03.927 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:03.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.926+0000 7f544058f640 1 Processor -- start 2024-03-25T17:33:03.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.926+0000 7f544058f640 1 -- start start 2024-03-25T17:33:03.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.926+0000 7f544058f640 10 monclient: init 2024-03-25T17:33:03.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:03.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:33:03.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 10 monclient(hunting): picked mon.noname-a con 0x7f54380a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 10 monclient(hunting): picked mon.noname-c con 0x7f543813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 10 monclient(hunting): picked mon.noname-b con 0x7f54381430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5438142520 con 0x7f54380a98f0 2024-03-25T17:33:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380b28a0 con 0x7f543813f710 2024-03-25T17:33:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380b2a30 con 0x7f54381430c0 2024-03-25T17:33:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.927+0000 7f544058f640 10 monclient(hunting): authenticate will time out at 5537.190231s 2024-03-25T17:33:03.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.935+0000 7f543e32b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f54380a98f0 0x7f54380a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:56902/0 (socket says 172.21.15.134:56902) 2024-03-25T17:33:03.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.935+0000 7f543e32b640 1 -- 172.21.15.134:0/3998326135 learned_addr learned my addr 172.21.15.134:0/3998326135 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:03.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.951+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223267455 0 0) 0x7f54380b2a30 con 0x7f54381430c0 2024-03-25T17:33:03.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.951+0000 7f543d329640 10 monclient(hunting): _negotiate 2024-03-25T17:33:03.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.951+0000 7f543d329640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:03.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.951+0000 7f543d329640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:03.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.951+0000 7f543d329640 10 monclient(hunting): my global_id is 34355 2024-03-25T17:33:03.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.951+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5424003520 con 0x7f54381430c0 2024-03-25T17:33:03.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.952+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685379302 0 0) 0x7f5438142520 con 0x7f54380a98f0 2024-03-25T17:33:03.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.952+0000 7f543d329640 10 monclient(hunting): _negotiate 2024-03-25T17:33:03.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.952+0000 7f543d329640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:03.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.952+0000 7f543d329640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:03.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.952+0000 7f543d329640 10 monclient(hunting): my global_id is 4368 2024-03-25T17:33:03.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.952+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54380b2a30 con 0x7f54380a98f0 2024-03-25T17:33:03.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.957+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752393509 0 0) 0x7f54380b28a0 con 0x7f543813f710 2024-03-25T17:33:03.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.957+0000 7f543d329640 10 monclient(hunting): _negotiate 2024-03-25T17:33:03.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.957+0000 7f543d329640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:03.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.957+0000 7f543d329640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:03.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.957+0000 7f543d329640 10 monclient(hunting): my global_id is 4387 2024-03-25T17:33:03.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.957+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5438142520 con 0x7f543813f710 2024-03-25T17:33:03.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 294544771 0 0) 0x7f5424003520 con 0x7f54381430c0 2024-03-25T17:33:03.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 >> v1:172.21.15.159:6789/0 conn(0x7f543813f710 legacy=0x7f54381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:03.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 >> v1:172.21.15.134:6789/0 conn(0x7f54380a98f0 legacy=0x7f54380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:03.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 10 monclient: _finish_hunting 0 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 1 monclient: found mon.noname-b 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54380b44c0 con 0x7f54381430c0 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 10 monclient: _finish_auth 0 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 10 monclient: _check_auth_tickets 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5430001a10 con 0x7f54381430c0 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f543d329640 10 monclient: dump: 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:03.961 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f544058f640 5 monclient: authenticate success, global_id 34355 2024-03-25T17:33:03.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.959+0000 7f544058f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:03.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5430001b70 con 0x7f54381430c0 2024-03-25T17:33:03.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f543d329640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:03.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f543d329640 1 -- 172.21.15.134:0/3998326135 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5430001f80 con 0x7f54381430c0 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f543d329640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f543d329640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f543d329640 10 monclient: dump: 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:03.978 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:03.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f544058f640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:03.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.977+0000 7f544058f640 4 set_mon_vals no callback set 2024-03-25T17:33:03.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.978+0000 7f544058f640 10 monclient: shutdown 2024-03-25T17:33:03.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.978+0000 7f544058f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:03.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.978+0000 7f544058f640 1 -- 172.21.15.134:0/3998326135 >> v1:172.21.15.134:6790/0 conn(0x7f54381430c0 legacy=0x7f543814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:03.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.978+0000 7f544058f640 1 -- 172.21.15.134:0/3998326135 shutdown_connections 2024-03-25T17:33:03.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.978+0000 7f544058f640 1 -- 172.21.15.134:0/3998326135 >> 172.21.15.134:0/3998326135 conn(0x7f54380b7ea0 msgr2=0x7f54380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:03.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.978+0000 7f544058f640 1 -- 172.21.15.134:0/3998326135 shutdown_connections 2024-03-25T17:33:03.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.979+0000 7f544058f640 1 -- 172.21.15.134:0/3998326135 wait complete. 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.979+0000 7f544058f640 10 monclient: build_initial_monmap 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 10 monclient: monmap: 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:03.981 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 1 librados: starting msgr at 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 1 librados: starting objecter 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 1 Processor -- start 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 1 -- start start 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 1 librados: setting wanted keys 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 1 librados: calling monclient init 2024-03-25T17:33:03.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.980+0000 7f544058f640 10 monclient: init 2024-03-25T17:33:03.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:03.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:33:03.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 10 monclient(hunting): picked mon.noname-c con 0x7f54380a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:03.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 10 monclient(hunting): picked mon.noname-a con 0x7f543813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:03.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 10 monclient(hunting): picked mon.noname-b con 0x7f54381430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:03.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5438136ad0 con 0x7f543813f710 2024-03-25T17:33:03.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54380b7520 con 0x7f54381430c0 2024-03-25T17:33:03.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5438135810 con 0x7f54380a98f0 2024-03-25T17:33:03.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:03.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.981+0000 7f544058f640 10 monclient(hunting): authenticate will time out at 5537.243969s 2024-03-25T17:33:03.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.986+0000 7f543eb2c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f54381430c0 0x7f5438130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34814/0 (socket says 172.21.15.134:34814) 2024-03-25T17:33:03.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.986+0000 7f543eb2c640 1 -- 172.21.15.134:0/2599917554 learned_addr learned my addr 172.21.15.134:0/2599917554 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:03.988 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.987+0000 7f542effd640 10 client.?.objecter ms_handle_connect 0x7f543813f710 2024-03-25T17:33:03.988 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.988+0000 7f542effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:03.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.990+0000 7f542effd640 10 client.?.objecter ms_handle_connect 0x7f54381430c0 2024-03-25T17:33:03.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.990+0000 7f542effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:03.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.992+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958233782 0 0) 0x7f5438136ad0 con 0x7f543813f710 2024-03-25T17:33:03.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.992+0000 7f542effd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:03.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.992+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:03.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.992+0000 7f542effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:03.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.992+0000 7f542effd640 10 monclient(hunting): my global_id is 4371 2024-03-25T17:33:03.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:03.992+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f541c004eb0 con 0x7f543813f710 2024-03-25T17:33:04.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.000+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859144953 0 0) 0x7f54380b7520 con 0x7f54381430c0 2024-03-25T17:33:04.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.001+0000 7f542effd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:04.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.001+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:04.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.001+0000 7f542effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:04.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.001+0000 7f542effd640 10 monclient(hunting): my global_id is 4390 2024-03-25T17:33:04.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.001+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5438136ad0 con 0x7f54381430c0 2024-03-25T17:33:04.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.001+0000 7f542effd640 10 client.?.objecter ms_handle_connect 0x7f54380a98f0 2024-03-25T17:33:04.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.001+0000 7f542effd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:04.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.013+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468723236 0 0) 0x7f5438135810 con 0x7f54380a98f0 2024-03-25T17:33:04.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.013+0000 7f542effd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:04.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.013+0000 7f542effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:04.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.013+0000 7f542effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:04.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.013+0000 7f542effd640 10 monclient(hunting): my global_id is 34358 2024-03-25T17:33:04.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.013+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54380b7520 con 0x7f54380a98f0 2024-03-25T17:33:04.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.021+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1696534060 0 0) 0x7f541c004eb0 con 0x7f543813f710 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.021+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5438135810 con 0x7f543813f710 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.021+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5428001930 con 0x7f543813f710 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.021+0000 7f542effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.021+0000 7f542effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.021+0000 7f542effd640 10 monclient(hunting): dump: 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:04.022 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:04.023 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:04.023 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:04.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.021+0000 7f542effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:04.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.035+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3255284290 0 0) 0x7f54380b7520 con 0x7f54380a98f0 2024-03-25T17:33:04.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.035+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f541c004eb0 con 0x7f54380a98f0 2024-03-25T17:33:04.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.035+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5434001930 con 0x7f54380a98f0 2024-03-25T17:33:04.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.035+0000 7f542effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:04.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.035+0000 7f542effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:04.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.035+0000 7f542effd640 10 monclient(hunting): dump: 2024-03-25T17:33:04.036 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:04.037 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 765880246 0 0) 0x7f5438135810 con 0x7f543813f710 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 >> v1:172.21.15.134:6790/0 conn(0x7f54380a98f0 legacy=0x7f5438135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 >> v1:172.21.15.159:6789/0 conn(0x7f54381430c0 legacy=0x7f5438130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 1 monclient: found mon.a 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54380b6180 con 0x7f543813f710 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 10 monclient: _finish_auth 0 2024-03-25T17:33:04.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f542effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 5 monclient: authenticate success, global_id 4371 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 10 monclient: _renew_subs 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54380b7a40 con 0x7f543813f710 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 10 monclient: _renew_subs 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5438131840 con 0x7f543813f710 2024-03-25T17:33:04.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.041+0000 7f544058f640 1 librados: init done 2024-03-25T17:33:04.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.042+0000 7f544058f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:04.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.042+0000 7f544058f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:04.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.042+0000 7f544058f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:33:04.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.042+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5438136f60 con 0x7f543813f710 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5428003610 con 0x7f543813f710 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f542effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5428003780 con 0x7f543813f710 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f542effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f542effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f542effd640 10 monclient: dump: 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:04.293 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f543cb28640 4 set_mon_vals no callback set 2024-03-25T17:33:04.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.292+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f5428019750 con 0x7f543813f710 2024-03-25T17:33:04.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.293+0000 7f543e32b640 10 monclient: get_auth_request con 0x7f541c06df40 auth_method 0 2024-03-25T17:33:04.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.293+0000 7f542effd640 10 client.4371.objecter ms_handle_connect 0x7f541c06df40 2024-03-25T17:33:04.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f54280534d0 con 0x7f543813f710 2024-03-25T17:33:04.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 10 client.4371.objecter ms_dispatch 0x7f5438000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:04.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:04.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:04.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T17:33:04.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f542804ee10 con 0x7f543813f710 2024-03-25T17:33:04.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:04.303 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.301+0000 7f542effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:04.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.525+0000 7f544058f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:04.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.526+0000 7f544058f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:04.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.526+0000 7f544058f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:33:04.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.526+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54380b6310 con 0x7f543813f710 2024-03-25T17:33:04.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.564+0000 7f542effd640 1 -- 172.21.15.134:0/2599917554 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f5428050f10 con 0x7f543813f710 2024-03-25T17:33:04.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.564+0000 7f542effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:04.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.564+0000 7f542effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:04.565 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:04.566 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:04.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 10 librados: watch_flush enter 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 10 librados: watch_flush exit 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 10 client.4371.objecter successfully canceled tick 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 >> v1:172.21.15.134:6800/610837766 conn(0x7f541c06df40 legacy=0x7f541c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 10 monclient: shutdown 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 >> v1:172.21.15.134:6789/0 conn(0x7f543813f710 legacy=0x7f54380b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 shutdown_connections 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 >> 172.21.15.134:0/2599917554 conn(0x7f54380b7ea0 msgr2=0x7f5438074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 shutdown_connections 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 1 -- 172.21.15.134:0/2599917554 wait complete. 2024-03-25T17:33:04.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:04.567+0000 7f544058f640 1 librados: shutdown 2024-03-25T17:33:04.578 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:04.621 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:09.637 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.631 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:09.647 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.668 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:09.685 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.682 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:09.699 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.692 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:09.721 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.693 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:09.709 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.696 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:09.725 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.727 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:09.755 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:04.730 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:09.746 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:11.384 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:11.501 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.501+0000 7fd7ffd32640 10 monclient: get_monmap_and_config 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.501+0000 7fd7ffd32640 10 monclient: build_initial_monmap 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.501+0000 7fd7ffd32640 10 monclient: monmap: 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:11.502226+0000 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:11.502226+0000 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:11.502 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:11.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.501+0000 7fd7ffd32640 1 Processor -- start 2024-03-25T17:33:11.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.502+0000 7fd7ffd32640 1 -- start start 2024-03-25T17:33:11.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.502+0000 7fd7ffd32640 10 monclient: init 2024-03-25T17:33:11.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.502+0000 7fd7ffd32640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:11.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.502+0000 7fd7ffd32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f8142520 con 0x7fd7f80a98f0 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f80b28a0 con 0x7fd7f813f710 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f80b2a30 con 0x7fd7f81430c0 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:11.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.503+0000 7fd7ffd32640 10 monclient(hunting): authenticate will time out at 5544.765831s 2024-03-25T17:33:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.520+0000 7fd7fdace640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fd7f80a98f0 0x7fd7f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:53530/0 (socket says 172.21.15.134:53530) 2024-03-25T17:33:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.520+0000 7fd7fdace640 1 -- 172.21.15.134:0/2612760272 learned_addr learned my addr 172.21.15.134:0/2612760272 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:11.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.533+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46312534 0 0) 0x7fd7f8142520 con 0x7fd7f80a98f0 2024-03-25T17:33:11.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.533+0000 7fd7fcacc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:11.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.533+0000 7fd7fcacc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:11.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.533+0000 7fd7fcacc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:11.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.533+0000 7fd7fcacc640 10 monclient(hunting): my global_id is 4374 2024-03-25T17:33:11.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.533+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7e4003520 con 0x7fd7f80a98f0 2024-03-25T17:33:11.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.535+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890143344 0 0) 0x7fd7f80b28a0 con 0x7fd7f813f710 2024-03-25T17:33:11.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.535+0000 7fd7fcacc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:11.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.535+0000 7fd7fcacc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:11.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.535+0000 7fd7fcacc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:11.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.535+0000 7fd7fcacc640 10 monclient(hunting): my global_id is 4393 2024-03-25T17:33:11.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.536+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f8142520 con 0x7fd7f813f710 2024-03-25T17:33:11.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.540+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957265283 0 0) 0x7fd7f80b2a30 con 0x7fd7f81430c0 2024-03-25T17:33:11.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.540+0000 7fd7fcacc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:11.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.540+0000 7fd7fcacc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:11.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.540+0000 7fd7fcacc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:11.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.540+0000 7fd7fcacc640 10 monclient(hunting): my global_id is 34361 2024-03-25T17:33:11.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.540+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7f80b28a0 con 0x7fd7f81430c0 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3722224938 0 0) 0x7fd7e4003520 con 0x7fd7f80a98f0 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 >> v1:172.21.15.134:6790/0 conn(0x7fd7f81430c0 legacy=0x7fd7f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 >> v1:172.21.15.159:6789/0 conn(0x7fd7f813f710 legacy=0x7fd7f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 10 monclient: _finish_hunting 0 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 1 monclient: found mon.noname-a 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:33:11.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f80b44c0 con 0x7fd7f80a98f0 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 10 monclient: _finish_auth 0 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 10 monclient: _check_auth_tickets 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fd7f4001690 con 0x7fd7f80a98f0 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7ffd32640 5 monclient: authenticate success, global_id 4374 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7ffd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7fcacc640 10 monclient: dump: 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.546 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.547 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:11.547 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:11.547 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:11.547 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:11.547 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:11.547 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:11.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.544+0000 7fd7ffd32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:11.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7f40017f0 con 0x7fd7f80a98f0 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7fcacc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7fcacc640 1 -- 172.21.15.134:0/2612760272 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fd7f4001c00 con 0x7fd7f80a98f0 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7ffd32640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7ffd32640 4 set_mon_vals no callback set 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7fcacc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7fcacc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.558+0000 7fd7fcacc640 10 monclient: dump: 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.559 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:11.560 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:11.560 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:11.560 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:11.560 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:11.560 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:11.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.559+0000 7fd7ffd32640 10 monclient: shutdown 2024-03-25T17:33:11.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.559+0000 7fd7ffd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:11.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.559+0000 7fd7ffd32640 1 -- 172.21.15.134:0/2612760272 >> v1:172.21.15.134:6789/0 conn(0x7fd7f80a98f0 legacy=0x7fd7f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:11.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.559+0000 7fd7ffd32640 1 -- 172.21.15.134:0/2612760272 shutdown_connections 2024-03-25T17:33:11.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.559+0000 7fd7ffd32640 1 -- 172.21.15.134:0/2612760272 >> 172.21.15.134:0/2612760272 conn(0x7fd7f80b7ea0 msgr2=0x7fd7f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:11.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.559+0000 7fd7ffd32640 1 -- 172.21.15.134:0/2612760272 shutdown_connections 2024-03-25T17:33:11.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.560+0000 7fd7ffd32640 1 -- 172.21.15.134:0/2612760272 wait complete. 2024-03-25T17:33:11.562 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.560+0000 7fd7ffd32640 10 monclient: build_initial_monmap 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient: monmap: 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 librados: starting msgr at 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 librados: starting objecter 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 Processor -- start 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 -- start start 2024-03-25T17:33:11.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 librados: setting wanted keys 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 librados: calling monclient init 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient: init 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient(hunting): picked mon.noname-a con 0x7fd7f80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient(hunting): picked mon.noname-c con 0x7fd7f813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient(hunting): picked mon.noname-b con 0x7fd7f81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f81db050 con 0x7fd7f80a98f0 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f81dab70 con 0x7fd7f81430c0 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f81dad00 con 0x7fd7f813f710 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:11.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.561+0000 7fd7ffd32640 10 monclient(hunting): authenticate will time out at 5544.824752s 2024-03-25T17:33:11.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.569+0000 7fd7fe2cf640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fd7f81430c0 0x7fd7f81393b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:56876/0 (socket says 172.21.15.134:56876) 2024-03-25T17:33:11.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.569+0000 7fd7fe2cf640 1 -- 172.21.15.134:0/3884503338 learned_addr learned my addr 172.21.15.134:0/3884503338 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:11.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.570+0000 7fd7ee7fc640 10 client.?.objecter ms_handle_connect 0x7fd7f81430c0 2024-03-25T17:33:11.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.570+0000 7fd7ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:11.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.576+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172359379 0 0) 0x7fd7f81dab70 con 0x7fd7f81430c0 2024-03-25T17:33:11.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.576+0000 7fd7ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:11.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.576+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:11.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.576+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:11.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.576+0000 7fd7ee7fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T17:33:11.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.576+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc004eb0 con 0x7fd7f81430c0 2024-03-25T17:33:11.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.580+0000 7fd7ee7fc640 10 client.?.objecter ms_handle_connect 0x7fd7f80a98f0 2024-03-25T17:33:11.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.580+0000 7fd7ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:11.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.583+0000 7fd7ee7fc640 10 client.?.objecter ms_handle_connect 0x7fd7f813f710 2024-03-25T17:33:11.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.583+0000 7fd7ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.592+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2896231345 0 0) 0x7fd7dc004eb0 con 0x7fd7f81430c0 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.592+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7f81dab70 con 0x7fd7f81430c0 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.592+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fd7f0001c70 con 0x7fd7f81430c0 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.592+0000 7fd7ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.592+0000 7fd7ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.592+0000 7fd7ee7fc640 10 monclient(hunting): dump: 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.593 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:11.594 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:11.594 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:11.594 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:11.594 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:11.594 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:11.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.592+0000 7fd7ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:11.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.598+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851708049 0 0) 0x7fd7f81db050 con 0x7fd7f80a98f0 2024-03-25T17:33:11.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.598+0000 7fd7ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:11.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.598+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:11.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.598+0000 7fd7ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:11.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.598+0000 7fd7ee7fc640 10 monclient(hunting): my global_id is 4377 2024-03-25T17:33:11.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.598+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7dc004eb0 con 0x7fd7f80a98f0 2024-03-25T17:33:11.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 103319378 0 0) 0x7fd7f81dab70 con 0x7fd7f81430c0 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 >> v1:172.21.15.134:6790/0 conn(0x7fd7f813f710 legacy=0x7fd7f81d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 >> v1:172.21.15.134:6789/0 conn(0x7fd7f80a98f0 legacy=0x7fd7f81db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 1 monclient: found mon.b 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f80b70a0 con 0x7fd7f81430c0 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 10 monclient: _finish_auth 0 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 5 monclient: authenticate success, global_id 4396 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 10 monclient: _renew_subs 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:11.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f80b72f0 con 0x7fd7f81430c0 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 10 monclient: _renew_subs 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f81e32a0 con 0x7fd7f81430c0 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.606+0000 7fd7ffd32640 1 librados: init done 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.608+0000 7fd7ffd32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.608+0000 7fd7ffd32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:11.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.608+0000 7fd7ffd32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:11.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.608+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f80627b0 con 0x7fd7f81430c0 2024-03-25T17:33:11.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7f0003990 con 0x7fd7f81430c0 2024-03-25T17:33:11.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fd7f0003de0 con 0x7fd7f81430c0 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7ee7fc640 10 monclient: dump: 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:11.884 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:11.885 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:11.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7effff640 4 set_mon_vals no callback set 2024-03-25T17:33:11.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.883+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fd7f0004000 con 0x7fd7f81430c0 2024-03-25T17:33:11.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.884+0000 7fd7fdace640 10 monclient: get_auth_request con 0x7fd7dc06de20 auth_method 0 2024-03-25T17:33:11.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.884+0000 7fd7ee7fc640 10 client.4396.objecter ms_handle_connect 0x7fd7dc06de20 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fd7f0054ab0 con 0x7fd7f81430c0 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 10 client.4396.objecter ms_dispatch 0x7fd7f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7f00505c0 con 0x7fd7f81430c0 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:11.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:11.888+0000 7fd7ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:12.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.118+0000 7fd7ffd32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:12.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.118+0000 7fd7ffd32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:12.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.118+0000 7fd7ffd32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:12.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.118+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7f81db200 con 0x7fd7f81430c0 2024-03-25T17:33:12.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.143+0000 7fd7ee7fc640 1 -- 172.21.15.134:0/3884503338 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fd7f81db200 con 0x7fd7f81430c0 2024-03-25T17:33:12.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.143+0000 7fd7ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:12.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.143+0000 7fd7ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:12.144 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:12.144 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 10 librados: watch_flush enter 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 10 librados: watch_flush exit 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 10 client.4396.objecter successfully canceled tick 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 >> v1:172.21.15.134:6800/610837766 conn(0x7fd7dc06de20 legacy=0x7fd7dc0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 10 monclient: shutdown 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.145+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 >> v1:172.21.15.159:6789/0 conn(0x7fd7f81430c0 legacy=0x7fd7f81393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.146+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 shutdown_connections 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.146+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 >> 172.21.15.134:0/3884503338 conn(0x7fd7f80b7ea0 msgr2=0x7fd7f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.146+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 shutdown_connections 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.146+0000 7fd7ffd32640 1 -- 172.21.15.134:0/3884503338 wait complete. 2024-03-25T17:33:12.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:12.146+0000 7fd7ffd32640 1 librados: shutdown 2024-03-25T17:33:12.159 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:18.965 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:19.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.078+0000 7f8098655640 10 monclient: get_monmap_and_config 2024-03-25T17:33:19.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.078+0000 7f8098655640 10 monclient: build_initial_monmap 2024-03-25T17:33:19.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.078+0000 7f8098655640 10 monclient: monmap: 2024-03-25T17:33:19.079 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:19.079 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:19.079 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:19.079502+0000 2024-03-25T17:33:19.080 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:19.079502+0000 2024-03-25T17:33:19.080 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:19.080 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:19.080 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:19.080 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:19.080 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:19.080 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:19.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.079+0000 7f8098655640 1 Processor -- start 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.079+0000 7f8098655640 1 -- start start 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.079+0000 7f8098655640 10 monclient: init 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 10 monclient(hunting): picked mon.noname-a con 0x7f80900a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 10 monclient(hunting): picked mon.noname-c con 0x7f809013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 10 monclient(hunting): picked mon.noname-b con 0x7f80901430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8090142520 con 0x7f80900a98f0 2024-03-25T17:33:19.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900b28a0 con 0x7f809013f710 2024-03-25T17:33:19.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80900b2a30 con 0x7f80901430c0 2024-03-25T17:33:19.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:19.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.080+0000 7f8098655640 10 monclient(hunting): authenticate will time out at 5552.342999s 2024-03-25T17:33:19.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.086+0000 7f8095bf0640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f809013f710 0x7f80901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:33412/0 (socket says 172.21.15.134:33412) 2024-03-25T17:33:19.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.086+0000 7f8095bf0640 1 -- 172.21.15.134:0/396928025 learned_addr learned my addr 172.21.15.134:0/396928025 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:19.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.090+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204171660 0 0) 0x7f80900b28a0 con 0x7f809013f710 2024-03-25T17:33:19.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.090+0000 7f80953ef640 10 monclient(hunting): _negotiate 2024-03-25T17:33:19.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.090+0000 7f80953ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:19.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.090+0000 7f80953ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:19.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.090+0000 7f80953ef640 10 monclient(hunting): my global_id is 4399 2024-03-25T17:33:19.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.090+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f807c003520 con 0x7f809013f710 2024-03-25T17:33:19.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.106+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494767749 0 0) 0x7f80900b2a30 con 0x7f80901430c0 2024-03-25T17:33:19.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.106+0000 7f80953ef640 10 monclient(hunting): _negotiate 2024-03-25T17:33:19.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.106+0000 7f80953ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:19.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.106+0000 7f80953ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:19.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.106+0000 7f80953ef640 10 monclient(hunting): my global_id is 34367 2024-03-25T17:33:19.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.106+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80900b28a0 con 0x7f80901430c0 2024-03-25T17:33:19.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413153117 0 0) 0x7f8090142520 con 0x7f80900a98f0 2024-03-25T17:33:19.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient(hunting): _negotiate 2024-03-25T17:33:19.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:19.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:19.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient(hunting): my global_id is 4380 2024-03-25T17:33:19.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80900b2a30 con 0x7f80900a98f0 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2709726063 0 0) 0x7f807c003520 con 0x7f809013f710 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 >> v1:172.21.15.134:6790/0 conn(0x7f80901430c0 legacy=0x7f809014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 >> v1:172.21.15.134:6789/0 conn(0x7f80900a98f0 legacy=0x7f80900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient: _finish_hunting 0 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 monclient: found mon.noname-c 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80900b44c0 con 0x7f809013f710 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient: _finish_auth 0 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient: _check_auth_tickets 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8080001690 con 0x7f809013f710 2024-03-25T17:33:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f8098655640 5 monclient: authenticate success, global_id 4399 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f8098655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f80953ef640 10 monclient: dump: 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:19.118 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:19.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.115+0000 7f8098655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8080001870 con 0x7f809013f710 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f80953ef640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f80953ef640 1 -- 172.21.15.134:0/396928025 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8080001c80 con 0x7f809013f710 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f8098655640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f8098655640 4 set_mon_vals no callback set 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f80953ef640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f80953ef640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.121+0000 7f80953ef640 10 monclient: dump: 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.122 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:19.123 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:19.123 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:19.123 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:19.123 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:19.123 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.122+0000 7f8098655640 10 monclient: shutdown 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.122+0000 7f8098655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.122+0000 7f8098655640 1 -- 172.21.15.134:0/396928025 >> v1:172.21.15.159:6789/0 conn(0x7f809013f710 legacy=0x7f80901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.122+0000 7f8098655640 1 -- 172.21.15.134:0/396928025 shutdown_connections 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.122+0000 7f8098655640 1 -- 172.21.15.134:0/396928025 >> 172.21.15.134:0/396928025 conn(0x7f80900b7ea0 msgr2=0x7f80900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.122+0000 7f8098655640 1 -- 172.21.15.134:0/396928025 shutdown_connections 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.122+0000 7f8098655640 1 -- 172.21.15.134:0/396928025 wait complete. 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 10 monclient: build_initial_monmap 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 10 monclient: monmap: 2024-03-25T17:33:19.125 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 1 librados: starting msgr at 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 1 librados: starting objecter 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 1 Processor -- start 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 1 -- start start 2024-03-25T17:33:19.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 1 librados: setting wanted keys 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 1 librados: calling monclient init 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.123+0000 7f8098655640 10 monclient: init 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 10 monclient(hunting): picked mon.noname-c con 0x7f80900a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 10 monclient(hunting): picked mon.noname-b con 0x7f809013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 10 monclient(hunting): picked mon.noname-a con 0x7f80901430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8090139570 con 0x7f80901430c0 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8090137c80 con 0x7f809013f710 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8090138b40 con 0x7f80900a98f0 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:19.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.124+0000 7f8098655640 10 monclient(hunting): authenticate will time out at 5552.386972s 2024-03-25T17:33:19.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.137+0000 7f80963f1640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f80900a98f0 0x7f8090138430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33280/0 (socket says 172.21.15.134:33280) 2024-03-25T17:33:19.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.137+0000 7f80963f1640 1 -- 172.21.15.134:0/3093099285 learned_addr learned my addr 172.21.15.134:0/3093099285 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:19.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.142+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f80900a98f0 2024-03-25T17:33:19.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.142+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:19.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.144+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f809013f710 2024-03-25T17:33:19.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.144+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:19.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.150+0000 7f8086ffd640 10 client.?.objecter ms_handle_connect 0x7f80901430c0 2024-03-25T17:33:19.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.150+0000 7f8086ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:19.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.152+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296114024 0 0) 0x7f8090137c80 con 0x7f809013f710 2024-03-25T17:33:19.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.152+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:19.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.152+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:19.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.152+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:19.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.152+0000 7f8086ffd640 10 monclient(hunting): my global_id is 4402 2024-03-25T17:33:19.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.152+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8074004eb0 con 0x7f809013f710 2024-03-25T17:33:19.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.156+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302475938 0 0) 0x7f8090138b40 con 0x7f80900a98f0 2024-03-25T17:33:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.156+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.156+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.156+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.156+0000 7f8086ffd640 10 monclient(hunting): my global_id is 34370 2024-03-25T17:33:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.156+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8090137c80 con 0x7f80900a98f0 2024-03-25T17:33:19.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.161+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525598489 0 0) 0x7f8090139570 con 0x7f80901430c0 2024-03-25T17:33:19.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.161+0000 7f8086ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:19.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.161+0000 7f8086ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:19.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.161+0000 7f8086ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:19.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.161+0000 7f8086ffd640 10 monclient(hunting): my global_id is 4383 2024-03-25T17:33:19.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.161+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8090138b40 con 0x7f80901430c0 2024-03-25T17:33:19.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.177+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980514275 0 0) 0x7f8090137c80 con 0x7f80900a98f0 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.178+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8090139570 con 0x7f80900a98f0 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.178+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f808c0019f0 con 0x7f80900a98f0 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.178+0000 7f8086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.178+0000 7f8086ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.178+0000 7f8086ffd640 10 monclient(hunting): dump: 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:19.179 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:19.180 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:19.180 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:19.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.178+0000 7f8086ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.180+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 168777306 0 0) 0x7f8074004eb0 con 0x7f809013f710 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.180+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8090137c80 con 0x7f809013f710 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.180+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8080001b80 con 0x7f809013f710 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.180+0000 7f8086ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.180+0000 7f8086ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.180+0000 7f8086ffd640 10 monclient(hunting): dump: 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:19.181 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.182 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.182 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:19.182 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:19.182 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:19.182 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:19.182 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:19.182 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:19.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2445512194 0 0) 0x7f8090137c80 con 0x7f809013f710 2024-03-25T17:33:19.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 >> v1:172.21.15.134:6790/0 conn(0x7f80900a98f0 legacy=0x7f8090138430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:19.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 >> v1:172.21.15.134:6789/0 conn(0x7f80901430c0 legacy=0x7f8090133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 1 monclient: found mon.b 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80901397c0 con 0x7f809013f710 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 10 monclient: _finish_auth 0 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8086ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8098655640 5 monclient: authenticate success, global_id 4402 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8098655640 10 monclient: _renew_subs 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8098655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.200+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8090134e70 con 0x7f809013f710 2024-03-25T17:33:19.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:19.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 10 monclient: _renew_subs 2024-03-25T17:33:19.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:19.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80901d8390 con 0x7f809013f710 2024-03-25T17:33:19.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 1 librados: init done 2024-03-25T17:33:19.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:19.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:19.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:19.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.201+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80900627b0 con 0x7f809013f710 2024-03-25T17:33:19.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.505+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8080001d20 con 0x7f809013f710 2024-03-25T17:33:19.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.505+0000 7f8086ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:19.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.505+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8080004d90 con 0x7f809013f710 2024-03-25T17:33:19.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.505+0000 7f8086ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:19.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.505+0000 7f8086ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:19.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.505+0000 7f8086ffd640 10 monclient: dump: 2024-03-25T17:33:19.506 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:19.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.505+0000 7f8094bee640 4 set_mon_vals no callback set 2024-03-25T17:33:19.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.506+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f808001ad60 con 0x7f809013f710 2024-03-25T17:33:19.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.506+0000 7f80963f1640 10 monclient: get_auth_request con 0x7f807406df70 auth_method 0 2024-03-25T17:33:19.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.507+0000 7f8086ffd640 10 client.4402.objecter ms_handle_connect 0x7f807406df70 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f8080054930 con 0x7f809013f710 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 10 client.4402.objecter ms_dispatch 0x7f8090000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8080050440 con 0x7f809013f710 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:19.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.510+0000 7f8086ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:19.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.740+0000 7f8098655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:19.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.740+0000 7f8098655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:19.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.740+0000 7f8098655640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:19.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.740+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8090069f40 con 0x7f809013f710 2024-03-25T17:33:19.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.785+0000 7f8086ffd640 1 -- 172.21.15.134:0/3093099285 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f8080052370 con 0x7f809013f710 2024-03-25T17:33:19.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.785+0000 7f8086ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:19.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.785+0000 7f8086ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:19.786 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:19.786 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:19.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 10 librados: watch_flush enter 2024-03-25T17:33:19.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 10 librados: watch_flush exit 2024-03-25T17:33:19.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 10 client.4402.objecter successfully canceled tick 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 >> v1:172.21.15.134:6800/610837766 conn(0x7f807406df70 legacy=0x7f8074070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 10 monclient: shutdown 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 >> v1:172.21.15.159:6789/0 conn(0x7f809013f710 legacy=0x7f80901d7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 shutdown_connections 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.788+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 >> 172.21.15.134:0/3093099285 conn(0x7f80900b7ea0 msgr2=0x7f809013c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.789+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 shutdown_connections 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.789+0000 7f8098655640 1 -- 172.21.15.134:0/3093099285 wait complete. 2024-03-25T17:33:19.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:19.789+0000 7f8098655640 1 librados: shutdown 2024-03-25T17:33:19.801 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:26.608 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:26.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.725+0000 7f30c4e63640 10 monclient: get_monmap_and_config 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.725+0000 7f30c4e63640 10 monclient: build_initial_monmap 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.725+0000 7f30c4e63640 10 monclient: monmap: 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:26.726473+0000 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:26.726473+0000 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:26.726 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:26.727 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.726+0000 7f30c4e63640 1 Processor -- start 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.726+0000 7f30c4e63640 1 -- start start 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.726+0000 7f30c4e63640 10 monclient: init 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 10 monclient(hunting): picked mon.noname-b con 0x7f30c00705b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 10 monclient(hunting): picked mon.noname-c con 0x7f30c01402d0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 10 monclient(hunting): picked mon.noname-a con 0x7f30c0149ae0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00718c0 con 0x7f30c0149ae0 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00714c0 con 0x7f30c01402d0 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c0072b50 con 0x7f30c00705b0 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:26.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.727+0000 7f30c4e63640 10 monclient(hunting): authenticate will time out at 5559.990108s 2024-03-25T17:33:26.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.736+0000 7f30b7fff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f30c01402d0 0x7f30c0140730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:44274/0 (socket says 172.21.15.134:44274) 2024-03-25T17:33:26.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.736+0000 7f30b7fff640 1 -- 172.21.15.134:0/1096002740 learned_addr learned my addr 172.21.15.134:0/1096002740 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:26.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.743+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916680703 0 0) 0x7f30c00714c0 con 0x7f30c01402d0 2024-03-25T17:33:26.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.743+0000 7f30bdd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:33:26.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.743+0000 7f30bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:26.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.743+0000 7f30bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:26.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.743+0000 7f30bdd9b640 10 monclient(hunting): my global_id is 4405 2024-03-25T17:33:26.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.743+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac003520 con 0x7f30c01402d0 2024-03-25T17:33:26.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.752+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438503796 0 0) 0x7f30c00718c0 con 0x7f30c0149ae0 2024-03-25T17:33:26.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.752+0000 7f30bdd9b640 10 monclient(hunting): _negotiate 2024-03-25T17:33:26.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.752+0000 7f30bdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:26.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.752+0000 7f30bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:26.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.752+0000 7f30bdd9b640 10 monclient(hunting): my global_id is 4386 2024-03-25T17:33:26.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.752+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30c00714c0 con 0x7f30c0149ae0 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2261192373 0 0) 0x7f30ac003520 con 0x7f30c01402d0 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 >> v1:172.21.15.134:6790/0 conn(0x7f30c00705b0 legacy=0x7f30c013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 >> v1:172.21.15.134:6789/0 conn(0x7f30c0149ae0 legacy=0x7f30c014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 10 monclient: _finish_hunting 0 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 1 monclient: found mon.noname-c 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c013dd10 con 0x7f30c01402d0 2024-03-25T17:33:26.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 10 monclient: _finish_auth 0 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 10 monclient: _check_auth_tickets 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f30a8001690 con 0x7f30c01402d0 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30c4e63640 5 monclient: authenticate success, global_id 4405 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30c4e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30bdd9b640 10 monclient: dump: 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:26.757 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:26.758 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:26.758 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:26.758 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:26.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.755+0000 7f30c4e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a80017f0 con 0x7f30c01402d0 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30bdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30bdd9b640 1 -- 172.21.15.134:0/1096002740 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f30a8001c00 con 0x7f30c01402d0 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30bdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30bdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30bdd9b640 10 monclient: dump: 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:26.764 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30c4e63640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:26.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.763+0000 7f30c4e63640 4 set_mon_vals no callback set 2024-03-25T17:33:26.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.764+0000 7f30c4e63640 10 monclient: shutdown 2024-03-25T17:33:26.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.764+0000 7f30c4e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:26.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.764+0000 7f30c4e63640 1 -- 172.21.15.134:0/1096002740 >> v1:172.21.15.159:6789/0 conn(0x7f30c01402d0 legacy=0x7f30c0140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:26.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.764+0000 7f30c4e63640 1 -- 172.21.15.134:0/1096002740 shutdown_connections 2024-03-25T17:33:26.767 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.764+0000 7f30c4e63640 1 -- 172.21.15.134:0/1096002740 >> 172.21.15.134:0/1096002740 conn(0x7f30c00b5d10 msgr2=0x7f30c00b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.764+0000 7f30c4e63640 1 -- 172.21.15.134:0/1096002740 shutdown_connections 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.764+0000 7f30c4e63640 1 -- 172.21.15.134:0/1096002740 wait complete. 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 10 monclient: build_initial_monmap 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 10 monclient: monmap: 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:26.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 1 librados: starting msgr at 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 1 librados: starting objecter 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 1 Processor -- start 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 1 -- start start 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 1 librados: setting wanted keys 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 1 librados: calling monclient init 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.765+0000 7f30c4e63640 10 monclient: init 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 10 monclient(hunting): picked mon.noname-a con 0x7f30c00705b0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 10 monclient(hunting): picked mon.noname-c con 0x7f30c01402d0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 10 monclient(hunting): picked mon.noname-b con 0x7f30c0149ae0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c01340c0 con 0x7f30c00705b0 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c013dfd0 con 0x7f30c0149ae0 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c01349f0 con 0x7f30c01402d0 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:26.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.766+0000 7f30c4e63640 10 monclient(hunting): authenticate will time out at 5560.029625s 2024-03-25T17:33:26.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.783+0000 7f30b7fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f30c01402d0 0x7f30c0132f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:32780/0 (socket says 172.21.15.134:32780) 2024-03-25T17:33:26.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.783+0000 7f30b7fff640 1 -- 172.21.15.134:0/2698522196 learned_addr learned my addr 172.21.15.134:0/2698522196 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:26.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.783+0000 7f30b6ffd640 10 client.?.objecter ms_handle_connect 0x7f30c01402d0 2024-03-25T17:33:26.784 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.783+0000 7f30b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:26.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.791+0000 7f30b6ffd640 10 client.?.objecter ms_handle_connect 0x7f30c00705b0 2024-03-25T17:33:26.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.791+0000 7f30b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:26.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.794+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633889087 0 0) 0x7f30c01349f0 con 0x7f30c01402d0 2024-03-25T17:33:26.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.795+0000 7f30b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:26.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.795+0000 7f30b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:26.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.795+0000 7f30b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:26.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.795+0000 7f30b6ffd640 10 monclient(hunting): my global_id is 34376 2024-03-25T17:33:26.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.795+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a0004eb0 con 0x7f30c01402d0 2024-03-25T17:33:26.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.798+0000 7f30b6ffd640 10 client.?.objecter ms_handle_connect 0x7f30c0149ae0 2024-03-25T17:33:26.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.798+0000 7f30b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:26.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.803+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1099509853 0 0) 0x7f30a0004eb0 con 0x7f30c01402d0 2024-03-25T17:33:26.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.803+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30c01349f0 con 0x7f30c01402d0 2024-03-25T17:33:26.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.807+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131897495 0 0) 0x7f30c01340c0 con 0x7f30c00705b0 2024-03-25T17:33:26.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.807+0000 7f30b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:26.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.807+0000 7f30b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:26.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.807+0000 7f30b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:26.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.807+0000 7f30b6ffd640 10 monclient(hunting): my global_id is 4389 2024-03-25T17:33:26.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.807+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30a0004eb0 con 0x7f30c00705b0 2024-03-25T17:33:26.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f30a8001ca0 con 0x7f30c01402d0 2024-03-25T17:33:26.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:26.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:26.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient(hunting): dump: 2024-03-25T17:33:26.811 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:26.811 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 970777451 0 0) 0x7f30c01349f0 con 0x7f30c01402d0 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 >> v1:172.21.15.159:6789/0 conn(0x7f30c0149ae0 legacy=0x7f30c01e20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 >> v1:172.21.15.134:6789/0 conn(0x7f30c00705b0 legacy=0x7f30c013e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:26.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 1 monclient: found mon.c 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c013ebe0 con 0x7f30c01402d0 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient: _finish_auth 0 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 5 monclient: authenticate success, global_id 34376 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 10 monclient: _renew_subs 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c013ff70 con 0x7f30c01402d0 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 10 client.34376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 10 monclient: _renew_subs 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c0133820 con 0x7f30c01402d0 2024-03-25T17:33:26.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.810+0000 7f30c4e63640 1 librados: init done 2024-03-25T17:33:26.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.811+0000 7f30c4e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:26.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.811+0000 7f30c4e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:26.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.811+0000 7f30c4e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:26.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.811+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c013e160 con 0x7f30c01402d0 2024-03-25T17:33:26.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.834+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a80011f0 con 0x7f30c01402d0 2024-03-25T17:33:26.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.834+0000 7f30b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:26.834 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.834+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f30a8004b10 con 0x7f30c01402d0 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.834+0000 7f30b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.834+0000 7f30b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.834+0000 7f30b6ffd640 10 monclient: dump: 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:26.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.834+0000 7f30bd59a640 4 set_mon_vals no callback set 2024-03-25T17:33:26.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.835+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f30a801aba0 con 0x7f30c01402d0 2024-03-25T17:33:26.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.836+0000 7f30be59c640 10 monclient: get_auth_request con 0x7f30a006dd40 auth_method 0 2024-03-25T17:33:26.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.836+0000 7f30b6ffd640 10 client.34376.objecter ms_handle_connect 0x7f30a006dd40 2024-03-25T17:33:26.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.838+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f30a80546a0 con 0x7f30c01402d0 2024-03-25T17:33:26.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.838+0000 7f30b6ffd640 10 client.34376.objecter ms_dispatch 0x7f30c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:26.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.838+0000 7f30b6ffd640 3 client.34376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:26.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.838+0000 7f30b6ffd640 3 client.34376.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:26.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:26.838+0000 7f30b6ffd640 20 client.34376.objecter dump_active .. 0 homeless 2024-03-25T17:33:27.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.122+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30a804ffe0 con 0x7f30c01402d0 2024-03-25T17:33:27.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.122+0000 7f30b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:27.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.122+0000 7f30b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:27.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.362+0000 7f30c4e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:27.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.362+0000 7f30c4e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:27.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.362+0000 7f30c4e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:27.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.362+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30c00627b0 con 0x7f30c01402d0 2024-03-25T17:33:27.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.425+0000 7f30b6ffd640 1 -- 172.21.15.134:0/2698522196 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f30a80520e0 con 0x7f30c01402d0 2024-03-25T17:33:27.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.425+0000 7f30b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:27.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.425+0000 7f30b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:27.426 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:27.426 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.427+0000 7f30c4e63640 10 librados: watch_flush enter 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.427+0000 7f30c4e63640 10 librados: watch_flush exit 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.427+0000 7f30c4e63640 20 client.34376.objecter shutdown clearing up homeless session... 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.427+0000 7f30c4e63640 10 client.34376.objecter successfully canceled tick 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 >> v1:172.21.15.134:6800/610837766 conn(0x7f30a006dd40 legacy=0x7f30a0070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 10 monclient: shutdown 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 >> v1:172.21.15.134:6790/0 conn(0x7f30c01402d0 legacy=0x7f30c0132f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 shutdown_connections 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 >> 172.21.15.134:0/2698522196 conn(0x7f30c00b5d10 msgr2=0x7f30c013b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 shutdown_connections 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 1 -- 172.21.15.134:0/2698522196 wait complete. 2024-03-25T17:33:27.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:27.428+0000 7f30c4e63640 1 librados: shutdown 2024-03-25T17:33:27.441 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:34.248 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:34.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.361+0000 7f68b939d640 10 monclient: get_monmap_and_config 2024-03-25T17:33:34.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.361+0000 7f68b939d640 10 monclient: build_initial_monmap 2024-03-25T17:33:34.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.361+0000 7f68b939d640 10 monclient: monmap: 2024-03-25T17:33:34.362 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:34.362 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:34.362443+0000 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:34.362443+0000 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:34.363 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.362+0000 7f68b939d640 1 Processor -- start 2024-03-25T17:33:34.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.362+0000 7f68b939d640 1 -- start start 2024-03-25T17:33:34.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.362+0000 7f68b939d640 10 monclient: init 2024-03-25T17:33:34.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.362+0000 7f68b939d640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.362+0000 7f68b939d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.362+0000 7f68b939d640 10 monclient(hunting): picked mon.noname-a con 0x7f68b40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.363+0000 7f68b939d640 10 monclient(hunting): picked mon.noname-c con 0x7f68b413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.363+0000 7f68b939d640 10 monclient(hunting): picked mon.noname-b con 0x7f68b41430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.363+0000 7f68b939d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b4142520 con 0x7f68b40a98f0 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.363+0000 7f68b939d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b40b28a0 con 0x7f68b413f710 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.363+0000 7f68b939d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b40b2a30 con 0x7f68b41430c0 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.363+0000 7f68b939d640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:34.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.363+0000 7f68b939d640 10 monclient(hunting): authenticate will time out at 5567.625842s 2024-03-25T17:33:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.370+0000 7f68abfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f68b413f710 0x7f68b41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:33130/0 (socket says 172.21.15.134:33130) 2024-03-25T17:33:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.370+0000 7f68abfff640 1 -- 172.21.15.134:0/925035458 learned_addr learned my addr 172.21.15.134:0/925035458 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:34.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.376+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2585868708 0 0) 0x7f68b40b28a0 con 0x7f68b413f710 2024-03-25T17:33:34.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.376+0000 7f68b27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:34.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.376+0000 7f68b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:34.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.376+0000 7f68b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:34.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.376+0000 7f68b27fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T17:33:34.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.376+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0003520 con 0x7f68b413f710 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.385+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566906355 0 0) 0x7f68a0003520 con 0x7f68b413f710 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.385+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 >> v1:172.21.15.134:6790/0 conn(0x7f68b41430c0 legacy=0x7f68b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.385+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 >> v1:172.21.15.134:6789/0 conn(0x7f68b40a98f0 legacy=0x7f68b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 1 monclient: found mon.noname-c 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b40b44c0 con 0x7f68b413f710 2024-03-25T17:33:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 10 monclient: _finish_auth 0 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f689c001690 con 0x7f68b413f710 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b939d640 5 monclient: authenticate success, global_id 4411 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b27fc640 10 monclient: dump: 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.388 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.389 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:34.389 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:34.389 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:34.389 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:34.389 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:34.389 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.386+0000 7f68b939d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:34.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f689c001860 con 0x7f68b413f710 2024-03-25T17:33:34.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:34.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b27fc640 1 -- 172.21.15.134:0/925035458 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f689c001cb0 con 0x7f68b413f710 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b27fc640 10 monclient: dump: 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:34.394 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:34.395 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:34.395 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b939d640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:34.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b939d640 4 set_mon_vals no callback set 2024-03-25T17:33:34.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b939d640 10 monclient: shutdown 2024-03-25T17:33:34.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:34.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.393+0000 7f68b939d640 1 -- 172.21.15.134:0/925035458 >> v1:172.21.15.159:6789/0 conn(0x7f68b413f710 legacy=0x7f68b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.394+0000 7f68b939d640 1 -- 172.21.15.134:0/925035458 shutdown_connections 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.394+0000 7f68b939d640 1 -- 172.21.15.134:0/925035458 >> 172.21.15.134:0/925035458 conn(0x7f68b40b7ea0 msgr2=0x7f68b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.394+0000 7f68b939d640 1 -- 172.21.15.134:0/925035458 shutdown_connections 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.394+0000 7f68b939d640 1 -- 172.21.15.134:0/925035458 wait complete. 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.394+0000 7f68b939d640 10 monclient: build_initial_monmap 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.394+0000 7f68b939d640 10 monclient: monmap: 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:34.398 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 librados: starting msgr at 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 librados: starting objecter 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 Processor -- start 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 -- start start 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 librados: setting wanted keys 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 librados: calling monclient init 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient: init 2024-03-25T17:33:34.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient(hunting): picked mon.noname-a con 0x7f68b40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient(hunting): picked mon.noname-c con 0x7f68b413f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient(hunting): picked mon.noname-b con 0x7f68b41430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b41d77a0 con 0x7f68b40a98f0 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b41d8340 con 0x7f68b41430c0 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b40abb90 con 0x7f68b413f710 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:34.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.395+0000 7f68b939d640 10 monclient(hunting): authenticate will time out at 5567.658689s 2024-03-25T17:33:34.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.402+0000 7f68abfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f68b413f710 0x7f68b41d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:57442/0 (socket says 172.21.15.134:57442) 2024-03-25T17:33:34.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.402+0000 7f68abfff640 1 -- 172.21.15.134:0/36730906 learned_addr learned my addr 172.21.15.134:0/36730906 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:34.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.403+0000 7f68ab7fe640 10 client.?.objecter ms_handle_connect 0x7f68b413f710 2024-03-25T17:33:34.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.403+0000 7f68ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:34.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.410+0000 7f68ab7fe640 10 client.?.objecter ms_handle_connect 0x7f68b41430c0 2024-03-25T17:33:34.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.410+0000 7f68ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:34.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.416+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311884874 0 0) 0x7f68b40abb90 con 0x7f68b413f710 2024-03-25T17:33:34.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.417+0000 7f68ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:34.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.417+0000 7f68ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:34.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.417+0000 7f68ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:34.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.417+0000 7f68ab7fe640 10 monclient(hunting): my global_id is 34379 2024-03-25T17:33:34.417 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.417+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6894004eb0 con 0x7f68b413f710 2024-03-25T17:33:34.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.420+0000 7f68ab7fe640 10 client.?.objecter ms_handle_connect 0x7f68b40a98f0 2024-03-25T17:33:34.420 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.420+0000 7f68ab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:34.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.422+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1010471483 0 0) 0x7f68b41d8340 con 0x7f68b41430c0 2024-03-25T17:33:34.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.422+0000 7f68ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:34.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.422+0000 7f68ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:34.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.422+0000 7f68ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:34.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.422+0000 7f68ab7fe640 10 monclient(hunting): my global_id is 4414 2024-03-25T17:33:34.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.422+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b40abb90 con 0x7f68b41430c0 2024-03-25T17:33:34.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.429+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254515889 0 0) 0x7f68b41d77a0 con 0x7f68b40a98f0 2024-03-25T17:33:34.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.429+0000 7f68ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:34.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.429+0000 7f68ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:34.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.429+0000 7f68ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:34.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.429+0000 7f68ab7fe640 10 monclient(hunting): my global_id is 4395 2024-03-25T17:33:34.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.429+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b41d8340 con 0x7f68b40a98f0 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.432+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 804634708 0 0) 0x7f68b40abb90 con 0x7f68b41430c0 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.432+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68b41d77a0 con 0x7f68b41430c0 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.432+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f68ac001a70 con 0x7f68b41430c0 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.432+0000 7f68ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.432+0000 7f68ab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.432+0000 7f68ab7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:34.433 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.432+0000 7f68ab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.433+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1821213880 0 0) 0x7f6894004eb0 con 0x7f68b413f710 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.433+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68b40abb90 con 0x7f68b413f710 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.433+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f689c001b70 con 0x7f68b413f710 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.433+0000 7f68ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.433+0000 7f68ab7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.433+0000 7f68ab7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:34.435 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:34.436 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.441+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 702207955 0 0) 0x7f68b41d8340 con 0x7f68b40a98f0 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.441+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6894004eb0 con 0x7f68b40a98f0 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.441+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f68a4001830 con 0x7f68b40a98f0 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.441+0000 7f68ab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.441+0000 7f68ab7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.441+0000 7f68ab7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:34.442 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:34.443 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3714567634 0 0) 0x7f68b40abb90 con 0x7f68b413f710 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 >> v1:172.21.15.159:6789/0 conn(0x7f68b41430c0 legacy=0x7f68b41e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 >> v1:172.21.15.134:6789/0 conn(0x7f68b40a98f0 legacy=0x7f68b4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 1 monclient: found mon.c 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:34.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b4137870 con 0x7f68b413f710 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 10 monclient: _finish_auth 0 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 5 monclient: authenticate success, global_id 34379 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 10 monclient: _renew_subs 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b4137b20 con 0x7f68b413f710 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 10 client.34379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 10 monclient: _renew_subs 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:34.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b4135220 con 0x7f68b413f710 2024-03-25T17:33:34.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.452+0000 7f68b939d640 1 librados: init done 2024-03-25T17:33:34.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.453+0000 7f68b939d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:34.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.453+0000 7f68b939d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:34.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.453+0000 7f68b939d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:34.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.453+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b4074cc0 con 0x7f68b413f710 2024-03-25T17:33:34.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f689c004bf0 con 0x7f68b413f710 2024-03-25T17:33:34.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:34.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f689c004f00 con 0x7f68b413f710 2024-03-25T17:33:34.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:34.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:34.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68ab7fe640 10 monclient: dump: 2024-03-25T17:33:34.751 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68b1ffb640 4 set_mon_vals no callback set 2024-03-25T17:33:34.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.750+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f689c01aed0 con 0x7f68b413f710 2024-03-25T17:33:34.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.751+0000 7f68b2ffd640 10 monclient: get_auth_request con 0x7f689406e110 auth_method 0 2024-03-25T17:33:34.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.751+0000 7f68ab7fe640 10 client.34379.objecter ms_handle_connect 0x7f689406e110 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f689c054990 con 0x7f68b413f710 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 10 client.34379.objecter ms_dispatch 0x7f68b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 3 client.34379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 3 client.34379.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 20 client.34379.objecter dump_active .. 0 homeless 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f689c0502d0 con 0x7f68b413f710 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:34.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.756+0000 7f68ab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:34.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.984+0000 7f68b939d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:34.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.984+0000 7f68b939d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:34.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.984+0000 7f68b939d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:34.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:34.984+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68b40ab9b0 con 0x7f68b413f710 2024-03-25T17:33:35.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.026+0000 7f68ab7fe640 1 -- 172.21.15.134:0/36730906 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f689c0523d0 con 0x7f68b413f710 2024-03-25T17:33:35.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.026+0000 7f68ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:35.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.026+0000 7f68ab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:35.027 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:35.027 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 10 librados: watch_flush enter 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 10 librados: watch_flush exit 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 20 client.34379.objecter shutdown clearing up homeless session... 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 10 client.34379.objecter successfully canceled tick 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 >> v1:172.21.15.134:6800/610837766 conn(0x7f689406e110 legacy=0x7f68940705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 10 monclient: shutdown 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 >> v1:172.21.15.134:6790/0 conn(0x7f68b413f710 legacy=0x7f68b41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:35.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 shutdown_connections 2024-03-25T17:33:35.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 >> 172.21.15.134:0/36730906 conn(0x7f68b40b7ea0 msgr2=0x7f68b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:35.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 shutdown_connections 2024-03-25T17:33:35.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 1 -- 172.21.15.134:0/36730906 wait complete. 2024-03-25T17:33:35.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:35.029+0000 7f68b939d640 1 librados: shutdown 2024-03-25T17:33:35.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:41.848 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.960+0000 7f4da0888640 10 monclient: get_monmap_and_config 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.960+0000 7f4da0888640 10 monclient: build_initial_monmap 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.960+0000 7f4da0888640 10 monclient: monmap: 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:41.961633+0000 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:41.961633+0000 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:41.961 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:41.962 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:41.962 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:41.962 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:41.962 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.961+0000 7f4da0888640 1 Processor -- start 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.961+0000 7f4da0888640 1 -- start start 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.961+0000 7f4da0888640 10 monclient: init 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 10 monclient(hunting): picked mon.noname-a con 0x7f4d9c0b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 10 monclient(hunting): picked mon.noname-b con 0x7f4d9c0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 10 monclient(hunting): picked mon.noname-c con 0x7f4d9c14be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:41.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c0af940 con 0x7f4d9c0b4d60 2024-03-25T17:33:41.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c0b0a70 con 0x7f4d9c14be90 2024-03-25T17:33:41.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c0b0d50 con 0x7f4d9c0b1730 2024-03-25T17:33:41.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:41.964 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.962+0000 7f4da0888640 10 monclient(hunting): authenticate will time out at 5575.225200s 2024-03-25T17:33:41.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.966+0000 7f4d99d9b640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f4d9c0b1730 0x7f4d9c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:41796/0 (socket says 172.21.15.134:41796) 2024-03-25T17:33:41.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.966+0000 7f4d99d9b640 1 -- 172.21.15.134:0/1355056435 learned_addr learned my addr 172.21.15.134:0/1355056435 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:41.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.973+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662148214 0 0) 0x7f4d9c0b0d50 con 0x7f4d9c0b1730 2024-03-25T17:33:41.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.973+0000 7f4d9959a640 10 monclient(hunting): _negotiate 2024-03-25T17:33:41.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.973+0000 7f4d9959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:41.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.973+0000 7f4d9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:41.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.973+0000 7f4d9959a640 10 monclient(hunting): my global_id is 34382 2024-03-25T17:33:41.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.973+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d84003520 con 0x7f4d9c0b1730 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.980+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 170111747 0 0) 0x7f4d84003520 con 0x7f4d9c0b1730 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.980+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 >> v1:172.21.15.159:6789/0 conn(0x7f4d9c14be90 legacy=0x7f4d9c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 >> v1:172.21.15.134:6789/0 conn(0x7f4d9c0b4d60 legacy=0x7f4d9c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 10 monclient: _finish_hunting 0 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 1 monclient: found mon.noname-b 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c0b1030 con 0x7f4d9c0b1730 2024-03-25T17:33:41.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 10 monclient: _finish_auth 0 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 10 monclient: _check_auth_tickets 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d90001690 con 0x7f4d9c0b1730 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4da0888640 5 monclient: authenticate success, global_id 34382 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4da0888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4d9959a640 10 monclient: dump: 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:41.983 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:41.984 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:41.984 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:41.984 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:41.984 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:41.984 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:41.984 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:41.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.981+0000 7f4da0888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:41.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d900017f0 con 0x7f4d9c0b1730 2024-03-25T17:33:41.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4d9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:41.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4d9959a640 1 -- 172.21.15.134:0/1355056435 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d90001c00 con 0x7f4d9c0b1730 2024-03-25T17:33:41.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4da0888640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:41.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4da0888640 4 set_mon_vals no callback set 2024-03-25T17:33:41.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4d9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4d9959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.985+0000 7f4d9959a640 10 monclient: dump: 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:41.987 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.986+0000 7f4da0888640 10 monclient: shutdown 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.986+0000 7f4da0888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.986+0000 7f4da0888640 1 -- 172.21.15.134:0/1355056435 >> v1:172.21.15.134:6790/0 conn(0x7f4d9c0b1730 legacy=0x7f4d9c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.986+0000 7f4da0888640 1 -- 172.21.15.134:0/1355056435 shutdown_connections 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.986+0000 7f4da0888640 1 -- 172.21.15.134:0/1355056435 >> 172.21.15.134:0/1355056435 conn(0x7f4d9c0617a0 msgr2=0x7f4d9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.987+0000 7f4da0888640 1 -- 172.21.15.134:0/1355056435 shutdown_connections 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.987+0000 7f4da0888640 1 -- 172.21.15.134:0/1355056435 wait complete. 2024-03-25T17:33:41.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 10 monclient: build_initial_monmap 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 10 monclient: monmap: 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:41.990 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:41.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 1 librados: starting msgr at 2024-03-25T17:33:41.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 1 librados: starting objecter 2024-03-25T17:33:41.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 1 Processor -- start 2024-03-25T17:33:41.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 1 -- start start 2024-03-25T17:33:41.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 1 librados: setting wanted keys 2024-03-25T17:33:41.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 1 librados: calling monclient init 2024-03-25T17:33:41.991 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.988+0000 7f4da0888640 10 monclient: init 2024-03-25T17:33:41.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:41.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:33:41.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 10 monclient(hunting): picked mon.noname-a con 0x7f4d9c0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:41.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 10 monclient(hunting): picked mon.noname-b con 0x7f4d9c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:41.992 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 10 monclient(hunting): picked mon.noname-c con 0x7f4d9c14be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:41.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c143900 con 0x7f4d9c0b1730 2024-03-25T17:33:41.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c13a550 con 0x7f4d9c0b4d60 2024-03-25T17:33:41.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d9c13bc40 con 0x7f4d9c14be90 2024-03-25T17:33:41.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:41.993 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.989+0000 7f4da0888640 10 monclient(hunting): authenticate will time out at 5575.252257s 2024-03-25T17:33:41.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.994+0000 7f4d9a59c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4d9c0b1730 0x7f4d9c139e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34992/0 (socket says 172.21.15.134:34992) 2024-03-25T17:33:41.994 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.994+0000 7f4d9a59c640 1 -- 172.21.15.134:0/3192360219 learned_addr learned my addr 172.21.15.134:0/3192360219 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:41.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.994+0000 7f4d8affd640 10 client.?.objecter ms_handle_connect 0x7f4d9c0b1730 2024-03-25T17:33:41.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.994+0000 7f4d8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:42.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.999+0000 7f4d8affd640 10 client.?.objecter ms_handle_connect 0x7f4d9c14be90 2024-03-25T17:33:42.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:41.999+0000 7f4d8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:42.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.005+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1395785717 0 0) 0x7f4d9c143900 con 0x7f4d9c0b1730 2024-03-25T17:33:42.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.005+0000 7f4d8affd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:42.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.005+0000 7f4d8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:42.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.005+0000 7f4d8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:42.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.005+0000 7f4d8affd640 10 monclient(hunting): my global_id is 4401 2024-03-25T17:33:42.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.005+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d7c004eb0 con 0x7f4d9c0b1730 2024-03-25T17:33:42.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.008+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506704515 0 0) 0x7f4d9c13bc40 con 0x7f4d9c14be90 2024-03-25T17:33:42.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.008+0000 7f4d8affd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:42.009 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.008+0000 7f4d8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:42.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.008+0000 7f4d8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:42.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.008+0000 7f4d8affd640 10 monclient(hunting): my global_id is 34385 2024-03-25T17:33:42.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.008+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d9c143900 con 0x7f4d9c14be90 2024-03-25T17:33:42.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.009+0000 7f4d8affd640 10 client.?.objecter ms_handle_connect 0x7f4d9c0b4d60 2024-03-25T17:33:42.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.009+0000 7f4d8affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:42.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.016+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 152598672 0 0) 0x7f4d9c143900 con 0x7f4d9c14be90 2024-03-25T17:33:42.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.017+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d9c13bc40 con 0x7f4d9c14be90 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.017+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d94001b90 con 0x7f4d9c14be90 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.017+0000 7f4d8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.017+0000 7f4d8affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.017+0000 7f4d8affd640 10 monclient(hunting): dump: 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:42.018 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:42.019 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:42.019 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:42.019 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:42.019 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:42.019 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:42.019 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:42.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.017+0000 7f4d8affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:42.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.019+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322560329 0 0) 0x7f4d9c13a550 con 0x7f4d9c0b4d60 2024-03-25T17:33:42.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.019+0000 7f4d8affd640 10 monclient(hunting): _negotiate 2024-03-25T17:33:42.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.019+0000 7f4d8affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:42.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.019+0000 7f4d8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:42.020 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.019+0000 7f4d8affd640 10 monclient(hunting): my global_id is 4420 2024-03-25T17:33:42.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.019+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d9c143900 con 0x7f4d9c0b4d60 2024-03-25T17:33:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2633723700 0 0) 0x7f4d9c13bc40 con 0x7f4d9c14be90 2024-03-25T17:33:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 >> v1:172.21.15.159:6789/0 conn(0x7f4d9c0b4d60 legacy=0x7f4d9c12fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 >> v1:172.21.15.134:6789/0 conn(0x7f4d9c0b1730 legacy=0x7f4d9c139e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 10 monclient: _finish_hunting 0 2024-03-25T17:33:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 1 monclient: found mon.c 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d9c13bdd0 con 0x7f4d9c14be90 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 10 monclient: _finish_auth 0 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 10 monclient: _check_auth_tickets 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4d8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 5 monclient: authenticate success, global_id 34385 2024-03-25T17:33:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 10 monclient: _renew_subs 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d9c1391c0 con 0x7f4d9c14be90 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 10 client.34385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 10 monclient: _renew_subs 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9c130670 con 0x7f4d9c14be90 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.029+0000 7f4da0888640 1 librados: init done 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.030+0000 7f4da0888640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.030+0000 7f4da0888640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.030+0000 7f4da0888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.030+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d9c1d7a80 con 0x7f4d9c14be90 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.340+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d94002040 con 0x7f4d9c14be90 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.340+0000 7f4d8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.340+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4d94003b30 con 0x7f4d9c14be90 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.340+0000 7f4d8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.340+0000 7f4d8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.340+0000 7f4d8affd640 10 monclient: dump: 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:42.341 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:42.342 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:42.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.340+0000 7f4d98d99640 4 set_mon_vals no callback set 2024-03-25T17:33:42.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.341+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f4d94003d50 con 0x7f4d9c14be90 2024-03-25T17:33:42.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.341+0000 7f4d9a59c640 10 monclient: get_auth_request con 0x7f4d7c06de10 auth_method 0 2024-03-25T17:33:42.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.341+0000 7f4d8affd640 10 client.34385.objecter ms_handle_connect 0x7f4d7c06de10 2024-03-25T17:33:42.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f4d940537f0 con 0x7f4d9c14be90 2024-03-25T17:33:42.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 10 client.34385.objecter ms_dispatch 0x7f4d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:42.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 3 client.34385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:42.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 3 client.34385.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:42.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 20 client.34385.objecter dump_active .. 0 homeless 2024-03-25T17:33:42.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d9404f130 con 0x7f4d9c14be90 2024-03-25T17:33:42.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:42.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.347+0000 7f4d8affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:42.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.567+0000 7f4da0888640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:42.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.567+0000 7f4da0888640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:42.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.567+0000 7f4da0888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:42.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.567+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d9c0627b0 con 0x7f4d9c14be90 2024-03-25T17:33:42.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.594+0000 7f4d8affd640 1 -- 172.21.15.134:0/3192360219 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f4d94051230 con 0x7f4d9c14be90 2024-03-25T17:33:42.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.594+0000 7f4d8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:42.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.594+0000 7f4d8affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:42.595 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:42.595 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:42.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 10 librados: watch_flush enter 2024-03-25T17:33:42.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 10 librados: watch_flush exit 2024-03-25T17:33:42.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 20 client.34385.objecter shutdown clearing up homeless session... 2024-03-25T17:33:42.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 10 client.34385.objecter successfully canceled tick 2024-03-25T17:33:42.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 >> v1:172.21.15.134:6800/610837766 conn(0x7f4d7c06de10 legacy=0x7f4d7c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:42.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 10 monclient: shutdown 2024-03-25T17:33:42.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:42.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 >> v1:172.21.15.134:6790/0 conn(0x7f4d9c14be90 legacy=0x7f4d9c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:42.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 shutdown_connections 2024-03-25T17:33:42.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 >> 172.21.15.134:0/3192360219 conn(0x7f4d9c0617a0 msgr2=0x7f4d9c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:42.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 shutdown_connections 2024-03-25T17:33:42.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 1 -- 172.21.15.134:0/3192360219 wait complete. 2024-03-25T17:33:42.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:42.596+0000 7f4da0888640 1 librados: shutdown 2024-03-25T17:33:42.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:44.711 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:49.727 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.714 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:49.730 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.754 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:04:49.771 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.763 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:49.779 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.776 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:49.792 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.779 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:49.808 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.785 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:49.814 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.807 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:04:49.823 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:44.807 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:04:49.836 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:33:49.415 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.531+0000 7f424913b640 10 monclient: get_monmap_and_config 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.531+0000 7f424913b640 10 monclient: build_initial_monmap 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.531+0000 7f424913b640 10 monclient: monmap: 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:49.532920+0000 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:49.532920+0000 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:49.533 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:49.534 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:49.534 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:49.534 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:49.534 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.532+0000 7f424913b640 1 Processor -- start 2024-03-25T17:33:49.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.532+0000 7f424913b640 1 -- start start 2024-03-25T17:33:49.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.532+0000 7f424913b640 10 monclient: init 2024-03-25T17:33:49.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:49.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:33:49.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 10 monclient(hunting): picked mon.noname-c con 0x7f42440a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:49.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 10 monclient(hunting): picked mon.noname-a con 0x7f424413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:49.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 10 monclient(hunting): picked mon.noname-b con 0x7f42441430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:49.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4244142520 con 0x7f424413f710 2024-03-25T17:33:49.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42440b28a0 con 0x7f42440a98f0 2024-03-25T17:33:49.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42440b2a30 con 0x7f42441430c0 2024-03-25T17:33:49.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:49.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.533+0000 7f424913b640 10 monclient(hunting): authenticate will time out at 5582.796173s 2024-03-25T17:33:49.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.544+0000 7f424359e640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f42441430c0 0x7f424414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:52850/0 (socket says 172.21.15.134:52850) 2024-03-25T17:33:49.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.544+0000 7f424359e640 1 -- 172.21.15.134:0/1257825632 learned_addr learned my addr 172.21.15.134:0/1257825632 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:49.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.550+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228410377 0 0) 0x7f42440b2a30 con 0x7f42441430c0 2024-03-25T17:33:49.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.550+0000 7f4241d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:33:49.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.550+0000 7f4241d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:49.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.550+0000 7f4241d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:49.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.550+0000 7f4241d9b640 10 monclient(hunting): my global_id is 34388 2024-03-25T17:33:49.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.550+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f422c003520 con 0x7f42441430c0 2024-03-25T17:33:49.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.557+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833861212 0 0) 0x7f4244142520 con 0x7f424413f710 2024-03-25T17:33:49.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.557+0000 7f4241d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:33:49.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.557+0000 7f4241d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:49.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.557+0000 7f4241d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:49.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.557+0000 7f4241d9b640 10 monclient(hunting): my global_id is 4404 2024-03-25T17:33:49.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.557+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42440b2a30 con 0x7f424413f710 2024-03-25T17:33:49.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.561+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302914001 0 0) 0x7f42440b28a0 con 0x7f42440a98f0 2024-03-25T17:33:49.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.561+0000 7f4241d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:33:49.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.561+0000 7f4241d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:49.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.561+0000 7f4241d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:49.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.561+0000 7f4241d9b640 10 monclient(hunting): my global_id is 4423 2024-03-25T17:33:49.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.561+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4244142520 con 0x7f42440a98f0 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3206328282 0 0) 0x7f422c003520 con 0x7f42441430c0 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 >> v1:172.21.15.159:6789/0 conn(0x7f42440a98f0 legacy=0x7f42440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 >> v1:172.21.15.134:6789/0 conn(0x7f424413f710 legacy=0x7f42441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 1 monclient: found mon.noname-b 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42440b44c0 con 0x7f42441430c0 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 10 monclient: _finish_auth 0 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:33:49.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f423c001a10 con 0x7f42441430c0 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f4241d9b640 10 monclient: dump: 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:49.564 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:49.565 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f424913b640 5 monclient: authenticate success, global_id 34388 2024-03-25T17:33:49.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.562+0000 7f424913b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:49.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f423c001bf0 con 0x7f42441430c0 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f4241d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f4241d9b640 1 -- 172.21.15.134:0/1257825632 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f423c002000 con 0x7f42441430c0 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f4241d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f4241d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f4241d9b640 10 monclient: dump: 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:49.570 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:49.571 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:49.571 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:49.571 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:49.571 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f424913b640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:49.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.569+0000 7f424913b640 4 set_mon_vals no callback set 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.570+0000 7f424913b640 10 monclient: shutdown 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.570+0000 7f424913b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.570+0000 7f424913b640 1 -- 172.21.15.134:0/1257825632 >> v1:172.21.15.134:6790/0 conn(0x7f42441430c0 legacy=0x7f424414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.570+0000 7f424913b640 1 -- 172.21.15.134:0/1257825632 shutdown_connections 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.570+0000 7f424913b640 1 -- 172.21.15.134:0/1257825632 >> 172.21.15.134:0/1257825632 conn(0x7f42440b7ea0 msgr2=0x7f42440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.570+0000 7f424913b640 1 -- 172.21.15.134:0/1257825632 shutdown_connections 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.570+0000 7f424913b640 1 -- 172.21.15.134:0/1257825632 wait complete. 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 10 monclient: build_initial_monmap 2024-03-25T17:33:49.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 10 monclient: monmap: 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 1 librados: starting msgr at 2024-03-25T17:33:49.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 1 librados: starting objecter 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 1 Processor -- start 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 1 -- start start 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 1 librados: setting wanted keys 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 1 librados: calling monclient init 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.571+0000 7f424913b640 10 monclient: init 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 10 monclient(hunting): picked mon.noname-c con 0x7f42440a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 10 monclient(hunting): picked mon.noname-b con 0x7f424413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 10 monclient(hunting): picked mon.noname-a con 0x7f42441430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4244137ab0 con 0x7f42441430c0 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4244138bd0 con 0x7f424413f710 2024-03-25T17:33:49.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4244138e40 con 0x7f42440a98f0 2024-03-25T17:33:49.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:49.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.572+0000 7f424913b640 10 monclient(hunting): authenticate will time out at 5582.835191s 2024-03-25T17:33:49.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.573+0000 7f424259c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f424413f710 0x7f42441df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:37026/0 (socket says 172.21.15.134:37026) 2024-03-25T17:33:49.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.573+0000 7f424259c640 1 -- 172.21.15.134:0/2907814810 learned_addr learned my addr 172.21.15.134:0/2907814810 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:49.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.576+0000 7f422b7fe640 10 client.?.objecter ms_handle_connect 0x7f424413f710 2024-03-25T17:33:49.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.576+0000 7f422b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:49.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.582+0000 7f422b7fe640 10 client.?.objecter ms_handle_connect 0x7f42440a98f0 2024-03-25T17:33:49.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.582+0000 7f422b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:49.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.585+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371175463 0 0) 0x7f4244138bd0 con 0x7f424413f710 2024-03-25T17:33:49.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.585+0000 7f422b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:49.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.585+0000 7f422b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:49.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.585+0000 7f422b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:49.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.585+0000 7f422b7fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T17:33:49.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.585+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4230004eb0 con 0x7f424413f710 2024-03-25T17:33:49.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.587+0000 7f422b7fe640 10 client.?.objecter ms_handle_connect 0x7f42441430c0 2024-03-25T17:33:49.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.587+0000 7f422b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:49.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.588+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870753964 0 0) 0x7f4244138e40 con 0x7f42440a98f0 2024-03-25T17:33:49.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.588+0000 7f422b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:49.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.588+0000 7f422b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:49.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.588+0000 7f422b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:49.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.588+0000 7f422b7fe640 10 monclient(hunting): my global_id is 34391 2024-03-25T17:33:49.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.588+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4244138bd0 con 0x7f42440a98f0 2024-03-25T17:33:49.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.592+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225399685 0 0) 0x7f4244137ab0 con 0x7f42441430c0 2024-03-25T17:33:49.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.592+0000 7f422b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:33:49.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.592+0000 7f422b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:49.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.592+0000 7f422b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:49.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.592+0000 7f422b7fe640 10 monclient(hunting): my global_id is 4407 2024-03-25T17:33:49.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.592+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4244138e40 con 0x7f42441430c0 2024-03-25T17:33:49.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.598+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2579282385 0 0) 0x7f4244138bd0 con 0x7f42440a98f0 2024-03-25T17:33:49.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.598+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4244137ab0 con 0x7f42440a98f0 2024-03-25T17:33:49.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.598+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f42340018b0 con 0x7f42440a98f0 2024-03-25T17:33:49.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.598+0000 7f422b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:49.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.598+0000 7f422b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:33:49.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.598+0000 7f422b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:49.599 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.598+0000 7f422b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:49.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.601+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1335486176 0 0) 0x7f4230004eb0 con 0x7f424413f710 2024-03-25T17:33:49.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.601+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4244138bd0 con 0x7f424413f710 2024-03-25T17:33:49.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.601+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f42380019b0 con 0x7f424413f710 2024-03-25T17:33:49.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.601+0000 7f422b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:49.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.601+0000 7f422b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.601+0000 7f422b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:49.603 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.615+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2828821092 0 0) 0x7f4244138e40 con 0x7f42441430c0 2024-03-25T17:33:49.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.615+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4230004eb0 con 0x7f42441430c0 2024-03-25T17:33:49.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.615+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f423c001f00 con 0x7f42441430c0 2024-03-25T17:33:49.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.615+0000 7f422b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:49.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.615+0000 7f422b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.615+0000 7f422b7fe640 10 monclient(hunting): dump: 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:49.616 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 883145657 0 0) 0x7f4244137ab0 con 0x7f42440a98f0 2024-03-25T17:33:49.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 >> v1:172.21.15.159:6789/0 conn(0x7f424413f710 legacy=0x7f42441df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:49.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 >> v1:172.21.15.134:6789/0 conn(0x7f42441430c0 legacy=0x7f42441e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:49.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:33:49.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 1 monclient: found mon.c 2024-03-25T17:33:49.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:49.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4244139120 con 0x7f42440a98f0 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f422b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 5 monclient: authenticate success, global_id 34391 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 10 monclient: _renew_subs 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4244139400 con 0x7f42440a98f0 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 10 client.34391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 10 monclient: _renew_subs 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42440b7880 con 0x7f42440a98f0 2024-03-25T17:33:49.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.617+0000 7f424913b640 1 librados: init done 2024-03-25T17:33:49.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.618+0000 7f424913b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:49.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.618+0000 7f424913b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:49.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.618+0000 7f424913b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:49.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.618+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4244137810 con 0x7f42440a98f0 2024-03-25T17:33:49.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.830+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4234003610 con 0x7f42440a98f0 2024-03-25T17:33:49.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.830+0000 7f422b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:49.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.830+0000 7f424159a640 4 set_mon_vals no callback set 2024-03-25T17:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.915+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4234003920 con 0x7f42440a98f0 2024-03-25T17:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.915+0000 7f422b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:49.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.916+0000 7f422b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.916+0000 7f422b7fe640 10 monclient: dump: 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:49.917 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:49.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.916+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f42340198f0 con 0x7f42440a98f0 2024-03-25T17:33:49.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.917+0000 7f424259c640 10 monclient: get_auth_request con 0x7f423006e0f0 auth_method 0 2024-03-25T17:33:49.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.917+0000 7f422b7fe640 10 client.34391.objecter ms_handle_connect 0x7f423006e0f0 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f42340535f0 con 0x7f42440a98f0 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 10 client.34391.objecter ms_dispatch 0x7f4244000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 3 client.34391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 3 client.34391.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 20 client.34391.objecter dump_active .. 0 homeless 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f423404ef30 con 0x7f42440a98f0 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:49.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:49.925+0000 7f422b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:50.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.193+0000 7f424913b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:50.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.193+0000 7f424913b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:50.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.193+0000 7f424913b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:33:50.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.193+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42440627b0 con 0x7f42440a98f0 2024-03-25T17:33:50.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.219+0000 7f422b7fe640 1 -- 172.21.15.134:0/2907814810 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f4234051030 con 0x7f42440a98f0 2024-03-25T17:33:50.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.219+0000 7f422b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:50.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.219+0000 7f422b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:50.220 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:50.220 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.221+0000 7f424913b640 10 librados: watch_flush enter 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.221+0000 7f424913b640 10 librados: watch_flush exit 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 20 client.34391.objecter shutdown clearing up homeless session... 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 10 client.34391.objecter successfully canceled tick 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 >> v1:172.21.15.134:6800/610837766 conn(0x7f423006e0f0 legacy=0x7f42300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 10 monclient: shutdown 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 >> v1:172.21.15.134:6790/0 conn(0x7f42440a98f0 legacy=0x7f4244137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 shutdown_connections 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 >> 172.21.15.134:0/2907814810 conn(0x7f42440b7ea0 msgr2=0x7f4244074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 shutdown_connections 2024-03-25T17:33:50.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 1 -- 172.21.15.134:0/2907814810 wait complete. 2024-03-25T17:33:50.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:50.222+0000 7f424913b640 1 librados: shutdown 2024-03-25T17:33:50.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:33:57.043 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.160+0000 7fbaa9e93640 10 monclient: get_monmap_and_config 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.160+0000 7fbaa9e93640 10 monclient: build_initial_monmap 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.160+0000 7fbaa9e93640 10 monclient: monmap: 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:33:57.161625+0000 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:33:57.161625+0000 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:57.161 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:57.162 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:57.162 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:33:57.162 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:33:57.162 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.161+0000 7fbaa9e93640 1 Processor -- start 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.161+0000 7fbaa9e93640 1 -- start start 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.161+0000 7fbaa9e93640 10 monclient: init 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 10 monclient(hunting): picked mon.noname-b con 0x7fbaa40a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 10 monclient(hunting): picked mon.noname-a con 0x7fbaa413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:57.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa41430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:57.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa4142520 con 0x7fbaa413f710 2024-03-25T17:33:57.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa40b28a0 con 0x7fbaa41430c0 2024-03-25T17:33:57.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa40b2a30 con 0x7fbaa40a98f0 2024-03-25T17:33:57.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:57.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.162+0000 7fbaa9e93640 10 monclient(hunting): authenticate will time out at 5590.425299s 2024-03-25T17:33:57.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.165+0000 7fbaa3fff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fbaa41430c0 0x7fbaa414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:41966/0 (socket says 172.21.15.134:41966) 2024-03-25T17:33:57.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.165+0000 7fbaa3fff640 1 -- 172.21.15.134:0/1669410569 learned_addr learned my addr 172.21.15.134:0/1669410569 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:57.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.178+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808096434 0 0) 0x7fbaa40b28a0 con 0x7fbaa41430c0 2024-03-25T17:33:57.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.178+0000 7fbaa27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:57.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.178+0000 7fbaa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:57.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.178+0000 7fbaa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:57.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.178+0000 7fbaa27fc640 10 monclient(hunting): my global_id is 4429 2024-03-25T17:33:57.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.178+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba8c003520 con 0x7fbaa41430c0 2024-03-25T17:33:57.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.187+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554352845 0 0) 0x7fbaa40b2a30 con 0x7fbaa40a98f0 2024-03-25T17:33:57.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.187+0000 7fbaa27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:57.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.187+0000 7fbaa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:57.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.187+0000 7fbaa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:57.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.187+0000 7fbaa27fc640 10 monclient(hunting): my global_id is 34394 2024-03-25T17:33:57.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.187+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa40b28a0 con 0x7fbaa40a98f0 2024-03-25T17:33:57.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.193+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876170430 0 0) 0x7fbaa4142520 con 0x7fbaa413f710 2024-03-25T17:33:57.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.193+0000 7fbaa27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:33:57.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.193+0000 7fbaa27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:57.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.193+0000 7fbaa27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:57.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.193+0000 7fbaa27fc640 10 monclient(hunting): my global_id is 4410 2024-03-25T17:33:57.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.193+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa40b2a30 con 0x7fbaa413f710 2024-03-25T17:33:57.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1523891982 0 0) 0x7fbaa40b28a0 con 0x7fbaa40a98f0 2024-03-25T17:33:57.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 >> v1:172.21.15.159:6789/0 conn(0x7fbaa41430c0 legacy=0x7fbaa414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:57.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 >> v1:172.21.15.134:6789/0 conn(0x7fbaa413f710 legacy=0x7fbaa41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 1 monclient: found mon.noname-b 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa40b44c0 con 0x7fbaa40a98f0 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 10 monclient: _finish_auth 0 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa9e93640 5 monclient: authenticate success, global_id 34394 2024-03-25T17:33:57.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.198+0000 7fbaa9e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.199+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fba94001630 con 0x7fbaa40a98f0 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.199+0000 7fbaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.199+0000 7fbaa27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.199+0000 7fbaa27fc640 10 monclient: dump: 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:57.201 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:57.202 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:57.202 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:57.202 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:57.202 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.199+0000 7fbaa9e93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:33:57.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba94001880 con 0x7fbaa40a98f0 2024-03-25T17:33:57.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa27fc640 1 -- 172.21.15.134:0/1669410569 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fba94001c10 con 0x7fbaa40a98f0 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa9e93640 10 monclient: get_monmap_and_config success 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa9e93640 4 set_mon_vals no callback set 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa27fc640 10 monclient: dump: 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:57.210 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:57.211 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:57.211 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:57.211 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:57.211 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa9e93640 10 monclient: shutdown 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa9e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.209+0000 7fbaa9e93640 1 -- 172.21.15.134:0/1669410569 >> v1:172.21.15.134:6790/0 conn(0x7fbaa40a98f0 legacy=0x7fbaa40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.210+0000 7fbaa9e93640 1 -- 172.21.15.134:0/1669410569 shutdown_connections 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.210+0000 7fbaa9e93640 1 -- 172.21.15.134:0/1669410569 >> 172.21.15.134:0/1669410569 conn(0x7fbaa40b7ea0 msgr2=0x7fbaa40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.210+0000 7fbaa9e93640 1 -- 172.21.15.134:0/1669410569 shutdown_connections 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.210+0000 7fbaa9e93640 1 -- 172.21.15.134:0/1669410569 wait complete. 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.210+0000 7fbaa9e93640 10 monclient: build_initial_monmap 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.210+0000 7fbaa9e93640 10 monclient: monmap: 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:33:57.213 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.211+0000 7fbaa9e93640 1 librados: starting msgr at 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.211+0000 7fbaa9e93640 1 librados: starting objecter 2024-03-25T17:33:57.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.211+0000 7fbaa9e93640 1 Processor -- start 2024-03-25T17:33:57.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.211+0000 7fbaa9e93640 1 -- start start 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.211+0000 7fbaa9e93640 1 librados: setting wanted keys 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.211+0000 7fbaa9e93640 1 librados: calling monclient init 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.211+0000 7fbaa9e93640 10 monclient: init 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 10 monclient: _reopen_session rank -1 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 10 monclient(hunting): picked mon.noname-a con 0x7fbaa40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 10 monclient(hunting): picked mon.noname-c con 0x7fbaa413f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 10 monclient(hunting): picked mon.noname-b con 0x7fbaa41430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa40abf10 con 0x7fbaa40a98f0 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa41daac0 con 0x7fbaa41430c0 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbaa4139950 con 0x7fbaa413f710 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 10 monclient(hunting): _renew_subs 2024-03-25T17:33:57.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.212+0000 7fbaa9e93640 10 monclient(hunting): authenticate will time out at 5590.475046s 2024-03-25T17:33:57.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.220+0000 7fbaa37fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbaa40a98f0 0x7fbaa41dafe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49808/0 (socket says 172.21.15.134:49808) 2024-03-25T17:33:57.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.220+0000 7fbaa37fe640 1 -- 172.21.15.134:0/2762875669 learned_addr learned my addr 172.21.15.134:0/2762875669 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:33:57.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.223+0000 7fba8bfff640 10 client.?.objecter ms_handle_connect 0x7fbaa40a98f0 2024-03-25T17:33:57.223 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.223+0000 7fba8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:57.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.235+0000 7fba8bfff640 10 client.?.objecter ms_handle_connect 0x7fbaa41430c0 2024-03-25T17:33:57.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.235+0000 7fba8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:57.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.236+0000 7fba8bfff640 10 client.?.objecter ms_handle_connect 0x7fbaa413f710 2024-03-25T17:33:57.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.236+0000 7fba8bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:33:57.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.238+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264931304 0 0) 0x7fbaa40abf10 con 0x7fbaa40a98f0 2024-03-25T17:33:57.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): my global_id is 4413 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba90004eb0 con 0x7fbaa40a98f0 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791072457 0 0) 0x7fbaa41daac0 con 0x7fbaa41430c0 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 10 monclient(hunting): my global_id is 4432 2024-03-25T17:33:57.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.239+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa40abf10 con 0x7fbaa41430c0 2024-03-25T17:33:57.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.258+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506347817 0 0) 0x7fbaa4139950 con 0x7fbaa413f710 2024-03-25T17:33:57.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.258+0000 7fba8bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:33:57.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.258+0000 7fba8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:33:57.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.258+0000 7fba8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:33:57.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.258+0000 7fba8bfff640 10 monclient(hunting): my global_id is 34397 2024-03-25T17:33:57.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.258+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa41daac0 con 0x7fbaa413f710 2024-03-25T17:33:57.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.264+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 760774541 0 0) 0x7fbaa40abf10 con 0x7fbaa41430c0 2024-03-25T17:33:57.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.264+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaa4139950 con 0x7fbaa41430c0 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.264+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fba9c001a70 con 0x7fbaa41430c0 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.264+0000 7fba8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.264+0000 7fba8bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.264+0000 7fba8bfff640 10 monclient(hunting): dump: 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:57.265 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:57.266 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:57.266 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.264+0000 7fba8bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.271+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2979394493 0 0) 0x7fba90004eb0 con 0x7fbaa40a98f0 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.271+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbaa40abf10 con 0x7fbaa40a98f0 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.271+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fba94001c50 con 0x7fbaa40a98f0 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.271+0000 7fba8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.271+0000 7fba8bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.271+0000 7fba8bfff640 10 monclient(hunting): dump: 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.272 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.273 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:57.273 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:57.273 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:57.273 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:57.273 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:57.273 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.283+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377174878 0 0) 0x7fbaa41daac0 con 0x7fbaa413f710 2024-03-25T17:33:57.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.283+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba90004eb0 con 0x7fbaa413f710 2024-03-25T17:33:57.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.283+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fba98001830 con 0x7fbaa413f710 2024-03-25T17:33:57.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.283+0000 7fba8bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:57.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.283+0000 7fba8bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:33:57.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.283+0000 7fba8bfff640 10 monclient(hunting): dump: 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:57.285 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 836843011 0 0) 0x7fbaa4139950 con 0x7fbaa41430c0 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 >> v1:172.21.15.134:6790/0 conn(0x7fbaa413f710 legacy=0x7fbaa4134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 >> v1:172.21.15.134:6789/0 conn(0x7fbaa40a98f0 legacy=0x7fbaa41dafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 1 monclient: found mon.b 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa4135340 con 0x7fbaa41430c0 2024-03-25T17:33:57.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 10 monclient: _finish_auth 0 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fba8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 5 monclient: authenticate success, global_id 4432 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 10 monclient: _renew_subs 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa41db6f0 con 0x7fbaa41430c0 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 10 monclient: _renew_subs 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa4134970 con 0x7fbaa41430c0 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.284+0000 7fbaa9e93640 1 librados: init done 2024-03-25T17:33:57.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.285+0000 7fbaa9e93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:57.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.286+0000 7fbaa9e93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:57.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.286+0000 7fbaa9e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:57.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.286+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa40b3ef0 con 0x7fbaa41430c0 2024-03-25T17:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.564+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba9c001ea0 con 0x7fbaa41430c0 2024-03-25T17:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.564+0000 7fba8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.564+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fba9c004d00 con 0x7fbaa41430c0 2024-03-25T17:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.564+0000 7fba8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:33:57.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.564+0000 7fba8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.564+0000 7fba8bfff640 10 monclient: dump: 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:33:57.566 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.564+0000 7fbaa1ffb640 4 set_mon_vals no callback set 2024-03-25T17:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.565+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fba9c01acd0 con 0x7fbaa41430c0 2024-03-25T17:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.566+0000 7fbaa37fe640 10 monclient: get_auth_request con 0x7fba9006e0b0 auth_method 0 2024-03-25T17:33:57.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.566+0000 7fba8bfff640 10 client.4432.objecter ms_handle_connect 0x7fba9006e0b0 2024-03-25T17:33:57.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.570+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fba9c054900 con 0x7fbaa41430c0 2024-03-25T17:33:57.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.570+0000 7fba8bfff640 10 client.4432.objecter ms_dispatch 0x7fbaa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:33:57.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.570+0000 7fba8bfff640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:33:57.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.570+0000 7fba8bfff640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:33:57.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.570+0000 7fba8bfff640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T17:33:57.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.571+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba9c050410 con 0x7fbaa41430c0 2024-03-25T17:33:57.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.571+0000 7fba8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:33:57.572 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.571+0000 7fba8bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:33:57.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.815+0000 7fbaa9e93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:57.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.815+0000 7fbaa9e93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:57.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.815+0000 7fbaa9e93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:33:57.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.815+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaa41dac50 con 0x7fbaa41430c0 2024-03-25T17:33:57.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.850+0000 7fba8bfff640 1 -- 172.21.15.134:0/2762875669 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fba9c052340 con 0x7fbaa41430c0 2024-03-25T17:33:57.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.850+0000 7fba8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:33:57.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.850+0000 7fba8bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:33:57.851 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:33:57.852 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:33:57.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 10 librados: watch_flush enter 2024-03-25T17:33:57.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 10 librados: watch_flush exit 2024-03-25T17:33:57.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 10 client.4432.objecter successfully canceled tick 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 >> v1:172.21.15.134:6800/610837766 conn(0x7fba9006e0b0 legacy=0x7fba90070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 10 monclient: shutdown 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.852+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 >> v1:172.21.15.159:6789/0 conn(0x7fbaa41430c0 legacy=0x7fbaa41e2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.853+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 shutdown_connections 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.853+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 >> 172.21.15.134:0/2762875669 conn(0x7fbaa40b7ea0 msgr2=0x7fbaa4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.853+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 shutdown_connections 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.853+0000 7fbaa9e93640 1 -- 172.21.15.134:0/2762875669 wait complete. 2024-03-25T17:33:57.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:33:57.853+0000 7fbaa9e93640 1 librados: shutdown 2024-03-25T17:33:57.866 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:04.673 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:04.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.791+0000 7f42c419d640 10 monclient: get_monmap_and_config 2024-03-25T17:34:04.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.791+0000 7f42c419d640 10 monclient: build_initial_monmap 2024-03-25T17:34:04.792 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.791+0000 7f42c419d640 10 monclient: monmap: 2024-03-25T17:34:04.792 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:04.792808+0000 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:04.792808+0000 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:04.793 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.792+0000 7f42c419d640 1 Processor -- start 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.792+0000 7f42c419d640 1 -- start start 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.792+0000 7f42c419d640 10 monclient: init 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc13f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 10 monclient(hunting): picked mon.noname-a con 0x7f42bc1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc142520 con 0x7f42bc1430c0 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0b28a0 con 0x7f42bc13f710 2024-03-25T17:34:04.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc0b2a30 con 0x7f42bc0a98f0 2024-03-25T17:34:04.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:04.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.793+0000 7f42c419d640 10 monclient(hunting): authenticate will time out at 5598.056440s 2024-03-25T17:34:04.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.804+0000 7f42c273a640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f42bc1430c0 0x7f42bc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:50460/0 (socket says 172.21.15.134:50460) 2024-03-25T17:34:04.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.804+0000 7f42c273a640 1 -- 172.21.15.134:0/4177556600 learned_addr learned my addr 172.21.15.134:0/4177556600 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:04.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.825+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 19897171 0 0) 0x7f42bc0b2a30 con 0x7f42bc0a98f0 2024-03-25T17:34:04.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.825+0000 7f42c0f37640 10 monclient(hunting): _negotiate 2024-03-25T17:34:04.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.825+0000 7f42c0f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:04.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.825+0000 7f42c0f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:04.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.825+0000 7f42c0f37640 10 monclient(hunting): my global_id is 34400 2024-03-25T17:34:04.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.825+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a8003520 con 0x7f42bc0a98f0 2024-03-25T17:34:04.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.826+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250286982 0 0) 0x7f42bc142520 con 0x7f42bc1430c0 2024-03-25T17:34:04.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.826+0000 7f42c0f37640 10 monclient(hunting): _negotiate 2024-03-25T17:34:04.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.826+0000 7f42c0f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:04.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.826+0000 7f42c0f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:04.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.826+0000 7f42c0f37640 10 monclient(hunting): my global_id is 4416 2024-03-25T17:34:04.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.826+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc0b2a30 con 0x7f42bc1430c0 2024-03-25T17:34:04.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.830+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977681567 0 0) 0x7f42bc0b28a0 con 0x7f42bc13f710 2024-03-25T17:34:04.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.830+0000 7f42c0f37640 10 monclient(hunting): _negotiate 2024-03-25T17:34:04.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.830+0000 7f42c0f37640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:04.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.830+0000 7f42c0f37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:04.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.830+0000 7f42c0f37640 10 monclient(hunting): my global_id is 4435 2024-03-25T17:34:04.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.830+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc142520 con 0x7f42bc13f710 2024-03-25T17:34:04.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1581034633 0 0) 0x7f42a8003520 con 0x7f42bc0a98f0 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 >> v1:172.21.15.159:6789/0 conn(0x7f42bc13f710 legacy=0x7f42bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 >> v1:172.21.15.134:6789/0 conn(0x7f42bc1430c0 legacy=0x7f42bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 10 monclient: _finish_hunting 0 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 1 monclient: found mon.noname-b 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc0b44c0 con 0x7f42bc0a98f0 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 10 monclient: _finish_auth 0 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 10 monclient: _check_auth_tickets 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f42b8001690 con 0x7f42bc0a98f0 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:04.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c0f37640 10 monclient: dump: 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c419d640 5 monclient: authenticate success, global_id 34400 2024-03-25T17:34:04.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.835+0000 7f42c419d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42b80017f0 con 0x7f42bc0a98f0 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c0f37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c0f37640 1 -- 172.21.15.134:0/4177556600 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f42b8001c00 con 0x7f42bc0a98f0 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c0f37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c0f37640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c0f37640 10 monclient: dump: 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:04.851 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c419d640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:04.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.850+0000 7f42c419d640 4 set_mon_vals no callback set 2024-03-25T17:34:04.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.851+0000 7f42c419d640 10 monclient: shutdown 2024-03-25T17:34:04.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.851+0000 7f42c419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.851+0000 7f42c419d640 1 -- 172.21.15.134:0/4177556600 >> v1:172.21.15.134:6790/0 conn(0x7f42bc0a98f0 legacy=0x7f42bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.851+0000 7f42c419d640 1 -- 172.21.15.134:0/4177556600 shutdown_connections 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.851+0000 7f42c419d640 1 -- 172.21.15.134:0/4177556600 >> 172.21.15.134:0/4177556600 conn(0x7f42bc0b7ea0 msgr2=0x7f42bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.851+0000 7f42c419d640 1 -- 172.21.15.134:0/4177556600 shutdown_connections 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.851+0000 7f42c419d640 1 -- 172.21.15.134:0/4177556600 wait complete. 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.852+0000 7f42c419d640 10 monclient: build_initial_monmap 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.852+0000 7f42c419d640 10 monclient: monmap: 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:04.854 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 1 librados: starting msgr at 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 1 librados: starting objecter 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 1 Processor -- start 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 1 -- start start 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 1 librados: setting wanted keys 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 1 librados: calling monclient init 2024-03-25T17:34:04.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 10 monclient: init 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 10 monclient(hunting): picked mon.noname-c con 0x7f42bc0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 10 monclient(hunting): picked mon.noname-b con 0x7f42bc13f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 10 monclient(hunting): picked mon.noname-a con 0x7f42bc1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.853+0000 7f42c419d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc138b20 con 0x7f42bc1430c0 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.854+0000 7f42c419d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc137a80 con 0x7f42bc13f710 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.854+0000 7f42c419d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42bc139880 con 0x7f42bc0a98f0 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.854+0000 7f42c419d640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:04.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.854+0000 7f42c419d640 10 monclient(hunting): authenticate will time out at 5598.116819s 2024-03-25T17:34:04.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.865+0000 7f42c273a640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f42bc1430c0 0x7f42bc1e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:50516/0 (socket says 172.21.15.134:50516) 2024-03-25T17:34:04.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.865+0000 7f42c273a640 1 -- 172.21.15.134:0/1425335757 learned_addr learned my addr 172.21.15.134:0/1425335757 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:04.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.866+0000 7f42b27fc640 10 client.?.objecter ms_handle_connect 0x7f42bc1430c0 2024-03-25T17:34:04.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.866+0000 7f42b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:04.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.870+0000 7f42b27fc640 10 client.?.objecter ms_handle_connect 0x7f42bc0a98f0 2024-03-25T17:34:04.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.870+0000 7f42b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:04.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.875+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964679221 0 0) 0x7f42bc138b20 con 0x7f42bc1430c0 2024-03-25T17:34:04.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.875+0000 7f42b27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:04.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.875+0000 7f42b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:04.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.875+0000 7f42b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:04.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.875+0000 7f42b27fc640 10 monclient(hunting): my global_id is 4419 2024-03-25T17:34:04.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.875+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a0004eb0 con 0x7f42bc1430c0 2024-03-25T17:34:04.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.876+0000 7f42b27fc640 10 client.?.objecter ms_handle_connect 0x7f42bc13f710 2024-03-25T17:34:04.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.876+0000 7f42b27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:04.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.884+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301885219 0 0) 0x7f42bc137a80 con 0x7f42bc13f710 2024-03-25T17:34:04.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.884+0000 7f42b27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:04.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.884+0000 7f42b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:04.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.884+0000 7f42b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:04.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.884+0000 7f42b27fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T17:34:04.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.884+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42bc138b20 con 0x7f42bc13f710 2024-03-25T17:34:04.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.888+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2069816895 0 0) 0x7f42a0004eb0 con 0x7f42bc1430c0 2024-03-25T17:34:04.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.888+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42bc137a80 con 0x7f42bc1430c0 2024-03-25T17:34:04.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.888+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f42b4001c70 con 0x7f42bc1430c0 2024-03-25T17:34:04.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.888+0000 7f42b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:04.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.888+0000 7f42b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:04.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.888+0000 7f42b27fc640 10 monclient(hunting): dump: 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.888+0000 7f42b27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.889+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788458468 0 0) 0x7f42bc139880 con 0x7f42bc0a98f0 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.889+0000 7f42b27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:04.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.889+0000 7f42b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:04.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.889+0000 7f42b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:04.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.889+0000 7f42b27fc640 10 monclient(hunting): my global_id is 34403 2024-03-25T17:34:04.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.889+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a0004eb0 con 0x7f42bc0a98f0 2024-03-25T17:34:04.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2678862792 0 0) 0x7f42bc137a80 con 0x7f42bc1430c0 2024-03-25T17:34:04.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 >> v1:172.21.15.134:6790/0 conn(0x7f42bc0a98f0 legacy=0x7f42bc137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:04.901 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 >> v1:172.21.15.159:6789/0 conn(0x7f42bc13f710 legacy=0x7f42bc139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 1 monclient: found mon.a 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42bc139a10 con 0x7f42bc1430c0 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 10 monclient: _finish_auth 0 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 5 monclient: authenticate success, global_id 4419 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 10 monclient: _renew_subs 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42bc1dab00 con 0x7f42bc1430c0 2024-03-25T17:34:04.902 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 10 monclient: _renew_subs 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42bc13b4f0 con 0x7f42bc1430c0 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.900+0000 7f42c419d640 1 librados: init done 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.901+0000 7f42c419d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.902+0000 7f42c419d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.902+0000 7f42c419d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:04.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:04.902+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42bc1dae00 con 0x7f42bc1430c0 2024-03-25T17:34:05.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.196+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42b4004b60 con 0x7f42bc1430c0 2024-03-25T17:34:05.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.196+0000 7f42b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:05.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.196+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f42b4004e70 con 0x7f42bc1430c0 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.196+0000 7f42b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.196+0000 7f42b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.196+0000 7f42b27fc640 10 monclient: dump: 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:05.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.196+0000 7f42b3fff640 4 set_mon_vals no callback set 2024-03-25T17:34:05.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.197+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f42b401ae40 con 0x7f42bc1430c0 2024-03-25T17:34:05.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.198+0000 7f42c1f39640 10 monclient: get_auth_request con 0x7f42a006de00 auth_method 0 2024-03-25T17:34:05.199 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.198+0000 7f42b27fc640 10 client.4419.objecter ms_handle_connect 0x7f42a006de00 2024-03-25T17:34:05.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.203+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f42b4054ac0 con 0x7f42bc1430c0 2024-03-25T17:34:05.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.203+0000 7f42b27fc640 10 client.4419.objecter ms_dispatch 0x7f42bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:05.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.203+0000 7f42b27fc640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:05.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.203+0000 7f42b27fc640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:05.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.204+0000 7f42b27fc640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-25T17:34:05.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.204+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42b4050400 con 0x7f42bc1430c0 2024-03-25T17:34:05.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.204+0000 7f42b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:05.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.204+0000 7f42b27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:05.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.444+0000 7f42c419d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:05.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.444+0000 7f42c419d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:05.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.444+0000 7f42c419d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:05.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.444+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42bc0b3e80 con 0x7f42bc1430c0 2024-03-25T17:34:05.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.479+0000 7f42b27fc640 1 -- 172.21.15.134:0/1425335757 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f42b4052500 con 0x7f42bc1430c0 2024-03-25T17:34:05.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.479+0000 7f42b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:05.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.479+0000 7f42b27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:05.480 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:05.480 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:05.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 10 librados: watch_flush enter 2024-03-25T17:34:05.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 10 librados: watch_flush exit 2024-03-25T17:34:05.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-25T17:34:05.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 10 client.4419.objecter successfully canceled tick 2024-03-25T17:34:05.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 >> v1:172.21.15.134:6800/610837766 conn(0x7f42a006de00 legacy=0x7f42a00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:05.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 10 monclient: shutdown 2024-03-25T17:34:05.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:05.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 >> v1:172.21.15.134:6789/0 conn(0x7f42bc1430c0 legacy=0x7f42bc1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:05.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 shutdown_connections 2024-03-25T17:34:05.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 >> 172.21.15.134:0/1425335757 conn(0x7f42bc0b7ea0 msgr2=0x7f42bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:05.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 shutdown_connections 2024-03-25T17:34:05.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 1 -- 172.21.15.134:0/1425335757 wait complete. 2024-03-25T17:34:05.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:05.481+0000 7f42c419d640 1 librados: shutdown 2024-03-25T17:34:05.494 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:09.794 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:14.810 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.823 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:14.840 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.865 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:14.882 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.866 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:14.882 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.867 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:14.895 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.871 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:14.900 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.882 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:14.898 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.897 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:14.925 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:09.912 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:14.929 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:12.300 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:12.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.512+0000 7f989dca5640 10 monclient: get_monmap_and_config 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.512+0000 7f989dca5640 10 monclient: build_initial_monmap 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.512+0000 7f989dca5640 10 monclient: monmap: 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:12.513444+0000 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:12.513444+0000 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:12.513 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.513+0000 7f989dca5640 1 Processor -- start 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.513+0000 7f989dca5640 1 -- start start 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.513+0000 7f989dca5640 10 monclient: init 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 10 monclient(hunting): picked mon.noname-a con 0x7f98980a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 10 monclient(hunting): picked mon.noname-c con 0x7f989813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 10 monclient(hunting): picked mon.noname-b con 0x7f98981430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9898142520 con 0x7f98980a98f0 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98980b28a0 con 0x7f989813f710 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98980b2a30 con 0x7f98981430c0 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:12.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.514+0000 7f989dca5640 10 monclient(hunting): authenticate will time out at 5605.776941s 2024-03-25T17:34:12.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.524+0000 7f98977fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f98980a98f0 0x7f98980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58938/0 (socket says 172.21.15.134:58938) 2024-03-25T17:34:12.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.524+0000 7f98977fe640 1 -- 172.21.15.134:0/2756297358 learned_addr learned my addr 172.21.15.134:0/2756297358 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:12.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.534+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836592924 0 0) 0x7f98980b2a30 con 0x7f98981430c0 2024-03-25T17:34:12.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.534+0000 7f98967fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:12.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.534+0000 7f98967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:12.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.534+0000 7f98967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:12.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.534+0000 7f98967fc640 10 monclient(hunting): my global_id is 34406 2024-03-25T17:34:12.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.534+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9880003520 con 0x7f98981430c0 2024-03-25T17:34:12.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.538+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2702403826 0 0) 0x7f98980b28a0 con 0x7f989813f710 2024-03-25T17:34:12.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.538+0000 7f98967fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:12.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.538+0000 7f98967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:12.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.538+0000 7f98967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:12.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.538+0000 7f98967fc640 10 monclient(hunting): my global_id is 4441 2024-03-25T17:34:12.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.538+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98980b2a30 con 0x7f989813f710 2024-03-25T17:34:12.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.540+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49190347 0 0) 0x7f9898142520 con 0x7f98980a98f0 2024-03-25T17:34:12.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.540+0000 7f98967fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:12.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.540+0000 7f98967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:12.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.540+0000 7f98967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:12.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.540+0000 7f98967fc640 10 monclient(hunting): my global_id is 4422 2024-03-25T17:34:12.541 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.540+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98980b28a0 con 0x7f98980a98f0 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4094335113 0 0) 0x7f98980b2a30 con 0x7f989813f710 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 >> v1:172.21.15.134:6790/0 conn(0x7f98981430c0 legacy=0x7f989814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 >> v1:172.21.15.134:6789/0 conn(0x7f98980a98f0 legacy=0x7f98980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 10 monclient: _finish_hunting 0 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 1 monclient: found mon.noname-c 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:34:12.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980b44c0 con 0x7f989813f710 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 10 monclient: _finish_auth 0 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 10 monclient: _check_auth_tickets 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f988c001670 con 0x7f989813f710 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f98967fc640 10 monclient: dump: 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:12.559 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:12.560 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:12.560 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:12.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f989dca5640 5 monclient: authenticate success, global_id 4441 2024-03-25T17:34:12.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.557+0000 7f989dca5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f988c0017d0 con 0x7f989813f710 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f98967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f98967fc640 1 -- 172.21.15.134:0/2756297358 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f988c001be0 con 0x7f989813f710 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f989dca5640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f989dca5640 4 set_mon_vals no callback set 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f98967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f98967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:12.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.582+0000 7f98967fc640 10 monclient: dump: 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:12.584 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:12.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.583+0000 7f989dca5640 10 monclient: shutdown 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.583+0000 7f989dca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.583+0000 7f989dca5640 1 -- 172.21.15.134:0/2756297358 >> v1:172.21.15.159:6789/0 conn(0x7f989813f710 legacy=0x7f98981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.583+0000 7f989dca5640 1 -- 172.21.15.134:0/2756297358 shutdown_connections 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.583+0000 7f989dca5640 1 -- 172.21.15.134:0/2756297358 >> 172.21.15.134:0/2756297358 conn(0x7f98980b7ea0 msgr2=0x7f98980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.584+0000 7f989dca5640 1 -- 172.21.15.134:0/2756297358 shutdown_connections 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.584+0000 7f989dca5640 1 -- 172.21.15.134:0/2756297358 wait complete. 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.584+0000 7f989dca5640 10 monclient: build_initial_monmap 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.584+0000 7f989dca5640 10 monclient: monmap: 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:12.586 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.585+0000 7f989dca5640 1 librados: starting msgr at 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.585+0000 7f989dca5640 1 librados: starting objecter 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.585+0000 7f989dca5640 1 Processor -- start 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.585+0000 7f989dca5640 1 -- start start 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.585+0000 7f989dca5640 1 librados: setting wanted keys 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.585+0000 7f989dca5640 1 librados: calling monclient init 2024-03-25T17:34:12.587 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.585+0000 7f989dca5640 10 monclient: init 2024-03-25T17:34:12.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:12.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:34:12.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 10 monclient(hunting): picked mon.noname-b con 0x7f98980a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:12.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 10 monclient(hunting): picked mon.noname-a con 0x7f989813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:12.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 10 monclient(hunting): picked mon.noname-c con 0x7f98981430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:12.588 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9898136510 con 0x7f989813f710 2024-03-25T17:34:12.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98980abb00 con 0x7f98980a98f0 2024-03-25T17:34:12.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98981da990 con 0x7f98981430c0 2024-03-25T17:34:12.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:12.589 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.586+0000 7f989dca5640 10 monclient(hunting): authenticate will time out at 5605.848946s 2024-03-25T17:34:12.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.593+0000 7f9896ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f989813f710 0x7f98981383a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58994/0 (socket says 172.21.15.134:58994) 2024-03-25T17:34:12.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.593+0000 7f9896ffd640 1 -- 172.21.15.134:0/2971784473 learned_addr learned my addr 172.21.15.134:0/2971784473 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:12.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.594+0000 7f987ffff640 10 client.?.objecter ms_handle_connect 0x7f989813f710 2024-03-25T17:34:12.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.594+0000 7f987ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:12.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.598+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520506090 0 0) 0x7f9898136510 con 0x7f989813f710 2024-03-25T17:34:12.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.598+0000 7f987ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:34:12.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.598+0000 7f987ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:12.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.598+0000 7f987ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:12.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.598+0000 7f987ffff640 10 monclient(hunting): my global_id is 4425 2024-03-25T17:34:12.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.598+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9884004eb0 con 0x7f989813f710 2024-03-25T17:34:12.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.599+0000 7f987ffff640 10 client.?.objecter ms_handle_connect 0x7f98980a98f0 2024-03-25T17:34:12.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.599+0000 7f987ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:12.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.604+0000 7f987ffff640 10 client.?.objecter ms_handle_connect 0x7f98981430c0 2024-03-25T17:34:12.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.604+0000 7f987ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:12.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.608+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2779585964 0 0) 0x7f9884004eb0 con 0x7f989813f710 2024-03-25T17:34:12.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.608+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9898136510 con 0x7f989813f710 2024-03-25T17:34:12.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.608+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f988c001670 con 0x7f989813f710 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.608+0000 7f987ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.608+0000 7f987ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.608+0000 7f987ffff640 10 monclient(hunting): dump: 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:12.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.608+0000 7f987ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:12.613 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.613+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191297969 0 0) 0x7f98980abb00 con 0x7f98980a98f0 2024-03-25T17:34:12.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.613+0000 7f987ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:34:12.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.613+0000 7f987ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:12.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.613+0000 7f987ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:12.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.613+0000 7f987ffff640 10 monclient(hunting): my global_id is 4444 2024-03-25T17:34:12.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.613+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9884004eb0 con 0x7f98980a98f0 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 320924716 0 0) 0x7f9898136510 con 0x7f989813f710 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 >> v1:172.21.15.134:6790/0 conn(0x7f98981430c0 legacy=0x7f98981e1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 >> v1:172.21.15.159:6789/0 conn(0x7f98980a98f0 legacy=0x7f98981daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 1 monclient: found mon.a 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898137820 con 0x7f989813f710 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 10 monclient: _finish_auth 0 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:34:12.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f987ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 5 monclient: authenticate success, global_id 4425 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 10 monclient: _renew_subs 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98981db5c0 con 0x7f989813f710 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 10 monclient: _renew_subs 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9898138c90 con 0x7f989813f710 2024-03-25T17:34:12.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.617+0000 7f989dca5640 1 librados: init done 2024-03-25T17:34:12.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.618+0000 7f989dca5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:12.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.618+0000 7f989dca5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:12.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.618+0000 7f989dca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:12.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.618+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98980b3ef0 con 0x7f989813f710 2024-03-25T17:34:12.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.918+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f988c0038a0 con 0x7f989813f710 2024-03-25T17:34:12.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.918+0000 7f987ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.918+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f988c003cb0 con 0x7f989813f710 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.918+0000 7f987ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.918+0000 7f987ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.918+0000 7f987ffff640 10 monclient: dump: 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:12.920 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:12.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.918+0000 7f9895ffb640 4 set_mon_vals no callback set 2024-03-25T17:34:12.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.919+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f988c003ed0 con 0x7f989813f710 2024-03-25T17:34:12.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.920+0000 7f98977fe640 10 monclient: get_auth_request con 0x7f988406de20 auth_method 0 2024-03-25T17:34:12.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.920+0000 7f987ffff640 10 client.4425.objecter ms_handle_connect 0x7f988406de20 2024-03-25T17:34:12.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.925+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f988c054950 con 0x7f989813f710 2024-03-25T17:34:12.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.925+0000 7f987ffff640 10 client.4425.objecter ms_dispatch 0x7f9898000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:12.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.925+0000 7f987ffff640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:12.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.925+0000 7f987ffff640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:12.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.925+0000 7f987ffff640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-25T17:34:12.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.925+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f988c050460 con 0x7f989813f710 2024-03-25T17:34:12.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.926+0000 7f987ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:12.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:12.926+0000 7f987ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:13.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.159+0000 7f989dca5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:13.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.159+0000 7f989dca5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:13.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.159+0000 7f989dca5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:13.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.159+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f989814f320 con 0x7f989813f710 2024-03-25T17:34:13.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.178+0000 7f987ffff640 1 -- 172.21.15.134:0/2971784473 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f988c052390 con 0x7f989813f710 2024-03-25T17:34:13.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.178+0000 7f987ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:13.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.178+0000 7f987ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:13.179 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:13.179 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:13.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.180+0000 7f989dca5640 10 librados: watch_flush enter 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.180+0000 7f989dca5640 10 librados: watch_flush exit 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.180+0000 7f989dca5640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.180+0000 7f989dca5640 10 client.4425.objecter successfully canceled tick 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 >> v1:172.21.15.134:6800/610837766 conn(0x7f988406de20 legacy=0x7f98840702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 10 monclient: shutdown 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 >> v1:172.21.15.134:6789/0 conn(0x7f989813f710 legacy=0x7f98981383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 shutdown_connections 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 >> 172.21.15.134:0/2971784473 conn(0x7f98980b7ea0 msgr2=0x7f9898074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 shutdown_connections 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 1 -- 172.21.15.134:0/2971784473 wait complete. 2024-03-25T17:34:13.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:13.181+0000 7f989dca5640 1 librados: shutdown 2024-03-25T17:34:13.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:20.001 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:20.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.113+0000 7ffbf6a74640 10 monclient: get_monmap_and_config 2024-03-25T17:34:20.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.113+0000 7ffbf6a74640 10 monclient: build_initial_monmap 2024-03-25T17:34:20.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.113+0000 7ffbf6a74640 10 monclient: monmap: 2024-03-25T17:34:20.114 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:20.114 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:20.114494+0000 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:20.114494+0000 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:20.115 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:20.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.114+0000 7ffbf6a74640 1 Processor -- start 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.114+0000 7ffbf6a74640 1 -- start start 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.114+0000 7ffbf6a74640 10 monclient: init 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 10 monclient(hunting): picked mon.noname-b con 0x7ffbf00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 10 monclient(hunting): picked mon.noname-a con 0x7ffbf013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 10 monclient(hunting): picked mon.noname-c con 0x7ffbf01430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf0142520 con 0x7ffbf013f710 2024-03-25T17:34:20.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf00b28a0 con 0x7ffbf01430c0 2024-03-25T17:34:20.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf00b2a30 con 0x7ffbf00a98f0 2024-03-25T17:34:20.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:20.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.115+0000 7ffbf6a74640 10 monclient(hunting): authenticate will time out at 5613.378445s 2024-03-25T17:34:20.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.118+0000 7ffbeffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ffbf013f710 0x7ffbf01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:59418/0 (socket says 172.21.15.134:59418) 2024-03-25T17:34:20.119 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.118+0000 7ffbeffff640 1 -- 172.21.15.134:0/3033998123 learned_addr learned my addr 172.21.15.134:0/3033998123 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:20.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.124+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612651983 0 0) 0x7ffbf0142520 con 0x7ffbf013f710 2024-03-25T17:34:20.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.124+0000 7ffbef7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:34:20.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.124+0000 7ffbef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:20.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.124+0000 7ffbef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:20.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.124+0000 7ffbef7fe640 10 monclient(hunting): my global_id is 4428 2024-03-25T17:34:20.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.124+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbe0003520 con 0x7ffbf013f710 2024-03-25T17:34:20.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986261949 0 0) 0x7ffbe0003520 con 0x7ffbf013f710 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 >> v1:172.21.15.134:6790/0 conn(0x7ffbf00a98f0 legacy=0x7ffbf00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 >> v1:172.21.15.159:6789/0 conn(0x7ffbf01430c0 legacy=0x7ffbf014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 1 monclient: found mon.noname-a 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf00b44c0 con 0x7ffbf013f710 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 10 monclient: _finish_auth 0 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:34:20.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ffbdc001690 con 0x7ffbf013f710 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbef7fe640 10 monclient: dump: 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:20.138 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:20.139 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:20.139 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:20.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbf6a74640 5 monclient: authenticate success, global_id 4428 2024-03-25T17:34:20.139 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.135+0000 7ffbf6a74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:20.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbdc0017f0 con 0x7ffbf013f710 2024-03-25T17:34:20.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:20.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbef7fe640 1 -- 172.21.15.134:0/3033998123 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ffbdc001c40 con 0x7ffbf013f710 2024-03-25T17:34:20.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbf6a74640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbf6a74640 4 set_mon_vals no callback set 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.146+0000 7ffbef7fe640 10 monclient: dump: 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:20.148 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:20.149 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:20.149 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:20.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.147+0000 7ffbf6a74640 10 monclient: shutdown 2024-03-25T17:34:20.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.147+0000 7ffbf6a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:20.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.147+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3033998123 >> v1:172.21.15.134:6789/0 conn(0x7ffbf013f710 legacy=0x7ffbf01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:20.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.147+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3033998123 shutdown_connections 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.147+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3033998123 >> 172.21.15.134:0/3033998123 conn(0x7ffbf00b7ea0 msgr2=0x7ffbf00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.147+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3033998123 shutdown_connections 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.147+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3033998123 wait complete. 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.148+0000 7ffbf6a74640 10 monclient: build_initial_monmap 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.148+0000 7ffbf6a74640 10 monclient: monmap: 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:20.152 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.149+0000 7ffbf6a74640 1 librados: starting msgr at 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.149+0000 7ffbf6a74640 1 librados: starting objecter 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.149+0000 7ffbf6a74640 1 Processor -- start 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.149+0000 7ffbf6a74640 1 -- start start 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.149+0000 7ffbf6a74640 1 librados: setting wanted keys 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.149+0000 7ffbf6a74640 1 librados: calling monclient init 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.149+0000 7ffbf6a74640 10 monclient: init 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:20.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 10 monclient(hunting): picked mon.noname-c con 0x7ffbf00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 10 monclient(hunting): picked mon.noname-b con 0x7ffbf013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 10 monclient(hunting): picked mon.noname-a con 0x7ffbf01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf00abac0 con 0x7ffbf01430c0 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf00abda0 con 0x7ffbf013f710 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbf01380e0 con 0x7ffbf00a98f0 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:20.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.150+0000 7ffbf6a74640 10 monclient(hunting): authenticate will time out at 5613.413223s 2024-03-25T17:34:20.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.153+0000 7ffbf4810640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7ffbf00a98f0 0x7ffbf00ac580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:37230/0 (socket says 172.21.15.134:37230) 2024-03-25T17:34:20.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.153+0000 7ffbf5011640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ffbf01430c0 0x7ffbf01e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:59452/0 (socket says 172.21.15.134:59452) 2024-03-25T17:34:20.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.153+0000 7ffbf4810640 1 -- 172.21.15.134:0/3725688967 learned_addr learned my addr 172.21.15.134:0/3725688967 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:20.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.154+0000 7ffbed7fa640 10 client.?.objecter ms_handle_connect 0x7ffbf00a98f0 2024-03-25T17:34:20.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.154+0000 7ffbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:20.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.155+0000 7ffbed7fa640 10 client.?.objecter ms_handle_connect 0x7ffbf01430c0 2024-03-25T17:34:20.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.155+0000 7ffbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:20.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.159+0000 7ffbed7fa640 10 client.?.objecter ms_handle_connect 0x7ffbf013f710 2024-03-25T17:34:20.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.159+0000 7ffbed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:20.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.168+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708432935 0 0) 0x7ffbf00abac0 con 0x7ffbf01430c0 2024-03-25T17:34:20.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.168+0000 7ffbed7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:20.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.168+0000 7ffbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:20.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.168+0000 7ffbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:20.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.168+0000 7ffbed7fa640 10 monclient(hunting): my global_id is 4431 2024-03-25T17:34:20.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.168+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbd4004eb0 con 0x7ffbf01430c0 2024-03-25T17:34:20.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.172+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761909123 0 0) 0x7ffbf00abda0 con 0x7ffbf013f710 2024-03-25T17:34:20.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.172+0000 7ffbed7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:20.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.172+0000 7ffbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:20.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.172+0000 7ffbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:20.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.172+0000 7ffbed7fa640 10 monclient(hunting): my global_id is 4450 2024-03-25T17:34:20.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.172+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbf00abac0 con 0x7ffbf013f710 2024-03-25T17:34:20.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.176+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622892899 0 0) 0x7ffbf01380e0 con 0x7ffbf00a98f0 2024-03-25T17:34:20.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.176+0000 7ffbed7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:34:20.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.176+0000 7ffbed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:20.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.176+0000 7ffbed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:20.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.176+0000 7ffbed7fa640 10 monclient(hunting): my global_id is 34415 2024-03-25T17:34:20.177 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.176+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbf00abda0 con 0x7ffbf00a98f0 2024-03-25T17:34:20.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.184+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 50895282 0 0) 0x7ffbf00abac0 con 0x7ffbf013f710 2024-03-25T17:34:20.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.184+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbf01380e0 con 0x7ffbf013f710 2024-03-25T17:34:20.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.185+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ffbdc001b00 con 0x7ffbf013f710 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.185+0000 7ffbed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.185+0000 7ffbed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.185+0000 7ffbed7fa640 10 monclient(hunting): dump: 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:20.186 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:20.187 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:20.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.185+0000 7ffbed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:20.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.188+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883079195 0 0) 0x7ffbd4004eb0 con 0x7ffbf01430c0 2024-03-25T17:34:20.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.188+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbf00abac0 con 0x7ffbf01430c0 2024-03-25T17:34:20.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.188+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ffbe8001bd0 con 0x7ffbf01430c0 2024-03-25T17:34:20.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.188+0000 7ffbed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:20.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.188+0000 7ffbed7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:20.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.188+0000 7ffbed7fa640 10 monclient(hunting): dump: 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:20.190 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:20.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1449801577 0 0) 0x7ffbf00abac0 con 0x7ffbf01430c0 2024-03-25T17:34:20.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 >> v1:172.21.15.134:6790/0 conn(0x7ffbf00a98f0 legacy=0x7ffbf00ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 >> v1:172.21.15.159:6789/0 conn(0x7ffbf013f710 legacy=0x7ffbf0138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 1 monclient: found mon.a 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbf0137d90 con 0x7ffbf01430c0 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 10 monclient: _finish_auth 0 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 5 monclient: authenticate success, global_id 4431 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 10 monclient: _renew_subs 2024-03-25T17:34:20.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:20.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbf0139240 con 0x7ffbf01430c0 2024-03-25T17:34:20.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:20.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 10 monclient: _renew_subs 2024-03-25T17:34:20.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:20.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbf0136830 con 0x7ffbf01430c0 2024-03-25T17:34:20.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.201+0000 7ffbf6a74640 1 librados: init done 2024-03-25T17:34:20.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.202+0000 7ffbf6a74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:20.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.202+0000 7ffbf6a74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:20.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.202+0000 7ffbf6a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:20.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.202+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbf00627b0 con 0x7ffbf01430c0 2024-03-25T17:34:20.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.503+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffbe80020c0 con 0x7ffbf01430c0 2024-03-25T17:34:20.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.503+0000 7ffbed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:20.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.504+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ffbe8003b30 con 0x7ffbf01430c0 2024-03-25T17:34:20.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.504+0000 7ffbed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:20.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.504+0000 7ffbed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.504+0000 7ffbed7fa640 10 monclient: dump: 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:20.506 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:20.507 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:20.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.504+0000 7ffbeeffd640 4 set_mon_vals no callback set 2024-03-25T17:34:20.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.504+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7ffbe8003d50 con 0x7ffbf01430c0 2024-03-25T17:34:20.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.505+0000 7ffbf4810640 10 monclient: get_auth_request con 0x7ffbd406dfb0 auth_method 0 2024-03-25T17:34:20.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.505+0000 7ffbed7fa640 10 client.4431.objecter ms_handle_connect 0x7ffbd406dfb0 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7ffbe8054a20 con 0x7ffbf01430c0 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 10 client.4431.objecter ms_dispatch 0x7ffbf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffbe8050360 con 0x7ffbf01430c0 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:20.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.510+0000 7ffbed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:20.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.728+0000 7ffbf6a74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:20.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.728+0000 7ffbf6a74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:20.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.728+0000 7ffbf6a74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:20.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.728+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffbf013f070 con 0x7ffbf01430c0 2024-03-25T17:34:20.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.749+0000 7ffbed7fa640 1 -- 172.21.15.134:0/3725688967 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7ffbe8052460 con 0x7ffbf01430c0 2024-03-25T17:34:20.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.749+0000 7ffbed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:20.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.749+0000 7ffbed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:20.751 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:20.751 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:20.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.751+0000 7ffbf6a74640 10 librados: watch_flush enter 2024-03-25T17:34:20.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 10 librados: watch_flush exit 2024-03-25T17:34:20.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-25T17:34:20.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 10 client.4431.objecter successfully canceled tick 2024-03-25T17:34:20.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 >> v1:172.21.15.134:6800/610837766 conn(0x7ffbd406dfb0 legacy=0x7ffbd4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:20.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 10 monclient: shutdown 2024-03-25T17:34:20.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:20.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 >> v1:172.21.15.134:6789/0 conn(0x7ffbf01430c0 legacy=0x7ffbf01e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:20.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 shutdown_connections 2024-03-25T17:34:20.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 >> 172.21.15.134:0/3725688967 conn(0x7ffbf00b7ea0 msgr2=0x7ffbf0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:20.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 shutdown_connections 2024-03-25T17:34:20.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 1 -- 172.21.15.134:0/3725688967 wait complete. 2024-03-25T17:34:20.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:20.752+0000 7ffbf6a74640 1 librados: shutdown 2024-03-25T17:34:20.767 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:27.573 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.690+0000 7f041c3ba640 10 monclient: get_monmap_and_config 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.690+0000 7f041c3ba640 10 monclient: build_initial_monmap 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.690+0000 7f041c3ba640 10 monclient: monmap: 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:27.691215+0000 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:27.691215+0000 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:27.691 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:27.692 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:27.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.691+0000 7f041c3ba640 1 Processor -- start 2024-03-25T17:34:27.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.691+0000 7f041c3ba640 1 -- start start 2024-03-25T17:34:27.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.691+0000 7f041c3ba640 10 monclient: init 2024-03-25T17:34:27.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 10 monclient(hunting): picked mon.noname-b con 0x7f04140a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 10 monclient(hunting): picked mon.noname-c con 0x7f041413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 10 monclient(hunting): picked mon.noname-a con 0x7f04141430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0414142520 con 0x7f04141430c0 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04140b28a0 con 0x7f041413f710 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04140b2a30 con 0x7f04140a98f0 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:27.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.692+0000 7f041c3ba640 10 monclient(hunting): authenticate will time out at 5620.955013s 2024-03-25T17:34:27.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.700+0000 7f041a156640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f04140a98f0 0x7f04140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:41878/0 (socket says 172.21.15.134:41878) 2024-03-25T17:34:27.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.700+0000 7f041a156640 1 -- 172.21.15.134:0/1454697880 learned_addr learned my addr 172.21.15.134:0/1454697880 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:27.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.707+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838055910 0 0) 0x7f04140b2a30 con 0x7f04140a98f0 2024-03-25T17:34:27.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.707+0000 7f0419154640 10 monclient(hunting): _negotiate 2024-03-25T17:34:27.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.707+0000 7f0419154640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:27.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.707+0000 7f0419154640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:27.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.707+0000 7f0419154640 10 monclient(hunting): my global_id is 34418 2024-03-25T17:34:27.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.707+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0400003520 con 0x7f04140a98f0 2024-03-25T17:34:27.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.708+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412544999 0 0) 0x7f04140b28a0 con 0x7f041413f710 2024-03-25T17:34:27.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.708+0000 7f0419154640 10 monclient(hunting): _negotiate 2024-03-25T17:34:27.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.708+0000 7f0419154640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:27.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.708+0000 7f0419154640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:27.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.708+0000 7f0419154640 10 monclient(hunting): my global_id is 4453 2024-03-25T17:34:27.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.708+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04140b2a30 con 0x7f041413f710 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1910314231 0 0) 0x7f04140b2a30 con 0x7f041413f710 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 >> v1:172.21.15.134:6790/0 conn(0x7f04140a98f0 legacy=0x7f04140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 >> v1:172.21.15.134:6789/0 conn(0x7f04141430c0 legacy=0x7f041414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 10 monclient: _finish_hunting 0 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 1 monclient: found mon.noname-c 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04140b44c0 con 0x7f041413f710 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 10 monclient: _finish_auth 0 2024-03-25T17:34:27.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 10 monclient: _check_auth_tickets 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0404001670 con 0x7f041413f710 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f0419154640 10 monclient: dump: 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:27.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f041c3ba640 5 monclient: authenticate success, global_id 4453 2024-03-25T17:34:27.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.715+0000 7f041c3ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:27.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04040017d0 con 0x7f041413f710 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f0419154640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f0419154640 1 -- 172.21.15.134:0/1454697880 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0404001c20 con 0x7f041413f710 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f041c3ba640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f041c3ba640 4 set_mon_vals no callback set 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f0419154640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f0419154640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.724+0000 7f0419154640 10 monclient: dump: 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:27.725 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:27.726 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:27.726 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.725+0000 7f041c3ba640 10 monclient: shutdown 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.725+0000 7f041c3ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.725+0000 7f041c3ba640 1 -- 172.21.15.134:0/1454697880 >> v1:172.21.15.159:6789/0 conn(0x7f041413f710 legacy=0x7f04141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.725+0000 7f041c3ba640 1 -- 172.21.15.134:0/1454697880 shutdown_connections 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.725+0000 7f041c3ba640 1 -- 172.21.15.134:0/1454697880 >> 172.21.15.134:0/1454697880 conn(0x7f04140b7ea0 msgr2=0x7f04140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.725+0000 7f041c3ba640 1 -- 172.21.15.134:0/1454697880 shutdown_connections 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.725+0000 7f041c3ba640 1 -- 172.21.15.134:0/1454697880 wait complete. 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 10 monclient: build_initial_monmap 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 10 monclient: monmap: 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:27.727 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 1 librados: starting msgr at 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 1 librados: starting objecter 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 1 Processor -- start 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 1 -- start start 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 1 librados: setting wanted keys 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 1 librados: calling monclient init 2024-03-25T17:34:27.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.726+0000 7f041c3ba640 10 monclient: init 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 10 monclient(hunting): picked mon.noname-b con 0x7f04140a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 10 monclient(hunting): picked mon.noname-a con 0x7f041413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 10 monclient(hunting): picked mon.noname-c con 0x7f04141430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04140ab8b0 con 0x7f041413f710 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0414138b90 con 0x7f04140a98f0 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0414138860 con 0x7f04141430c0 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:27.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.727+0000 7f041c3ba640 10 monclient(hunting): authenticate will time out at 5620.990344s 2024-03-25T17:34:27.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.738+0000 7f041a156640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f04140a98f0 0x7f0414137ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:56322/0 (socket says 172.21.15.134:56322) 2024-03-25T17:34:27.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.738+0000 7f041a156640 1 -- 172.21.15.134:0/3298116502 learned_addr learned my addr 172.21.15.134:0/3298116502 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:27.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.741+0000 7f040affd640 10 client.?.objecter ms_handle_connect 0x7f04140a98f0 2024-03-25T17:34:27.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.741+0000 7f040affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:27.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.752+0000 7f040affd640 10 client.?.objecter ms_handle_connect 0x7f041413f710 2024-03-25T17:34:27.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.752+0000 7f040affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:27.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155964737 0 0) 0x7f0414138b90 con 0x7f04140a98f0 2024-03-25T17:34:27.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:27.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:27.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:27.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 10 monclient(hunting): my global_id is 4456 2024-03-25T17:34:27.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f8004eb0 con 0x7f04140a98f0 2024-03-25T17:34:27.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 10 client.?.objecter ms_handle_connect 0x7f04141430c0 2024-03-25T17:34:27.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.753+0000 7f040affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:27.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.760+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865434048 0 0) 0x7f04140ab8b0 con 0x7f041413f710 2024-03-25T17:34:27.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.761+0000 7f040affd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:27.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.761+0000 7f040affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:27.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.761+0000 7f040affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:27.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.761+0000 7f040affd640 10 monclient(hunting): my global_id is 4434 2024-03-25T17:34:27.762 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.761+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0414138b90 con 0x7f041413f710 2024-03-25T17:34:27.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.770+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436924849 0 0) 0x7f0414138860 con 0x7f04141430c0 2024-03-25T17:34:27.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.770+0000 7f040affd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:27.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.770+0000 7f040affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:27.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.770+0000 7f040affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:27.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.770+0000 7f040affd640 10 monclient(hunting): my global_id is 34421 2024-03-25T17:34:27.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.770+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04140ab8b0 con 0x7f04141430c0 2024-03-25T17:34:27.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.771+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1516894683 0 0) 0x7f03f8004eb0 con 0x7f04140a98f0 2024-03-25T17:34:27.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.771+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0414138860 con 0x7f04140a98f0 2024-03-25T17:34:27.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.771+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0410001990 con 0x7f04140a98f0 2024-03-25T17:34:27.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.771+0000 7f040affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:27.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.771+0000 7f040affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:27.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.771+0000 7f040affd640 10 monclient(hunting): dump: 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:27.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.771+0000 7f040affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.782+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2656502069 0 0) 0x7f0414138b90 con 0x7f041413f710 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.782+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03f8004eb0 con 0x7f041413f710 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.782+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0404001b20 con 0x7f041413f710 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.782+0000 7f040affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.782+0000 7f040affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.782+0000 7f040affd640 10 monclient(hunting): dump: 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:27.783 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:27.784 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:27.784 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:27.784 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:27.784 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:27.784 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:27.793 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.792+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2076727010 0 0) 0x7f0414138860 con 0x7f04140a98f0 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.792+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 >> v1:172.21.15.134:6790/0 conn(0x7f04141430c0 legacy=0x7f0414135730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.792+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 >> v1:172.21.15.134:6789/0 conn(0x7f041413f710 legacy=0x7f0414135020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.792+0000 7f040affd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f040affd640 1 monclient: found mon.b 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f040affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f040affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f041414e6e0 con 0x7f04140a98f0 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f040affd640 10 monclient: _finish_auth 0 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f040affd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f040affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 5 monclient: authenticate success, global_id 4456 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 10 monclient: _renew_subs 2024-03-25T17:34:27.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f041414e990 con 0x7f04140a98f0 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 10 monclient: _renew_subs 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04141e2f20 con 0x7f04140a98f0 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.793+0000 7f041c3ba640 1 librados: init done 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.794+0000 7f041c3ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:27.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.794+0000 7f041c3ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:27.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.794+0000 7f041c3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:27.796 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:27.794+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04140627b0 con 0x7f04140a98f0 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.087+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04100047e0 con 0x7f04140a98f0 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.087+0000 7f040affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.087+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0410004af0 con 0x7f04140a98f0 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.087+0000 7f040affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.087+0000 7f040affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.087+0000 7f040affd640 10 monclient: dump: 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:28.088 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:28.089 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:28.089 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:28.089 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:28.089 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:28.089 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:28.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.087+0000 7f0418953640 4 set_mon_vals no callback set 2024-03-25T17:34:28.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.088+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f041001ab80 con 0x7f04140a98f0 2024-03-25T17:34:28.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.088+0000 7f0419955640 10 monclient: get_auth_request con 0x7f03f806dd70 auth_method 0 2024-03-25T17:34:28.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.089+0000 7f040affd640 10 client.4456.objecter ms_handle_connect 0x7f03f806dd70 2024-03-25T17:34:28.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.095+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f0410054870 con 0x7f04140a98f0 2024-03-25T17:34:28.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.095+0000 7f040affd640 10 client.4456.objecter ms_dispatch 0x7f0414000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:28.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.095+0000 7f040affd640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:28.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.095+0000 7f040affd640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:28.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.095+0000 7f040affd640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T17:34:28.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.096+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04100501b0 con 0x7f04140a98f0 2024-03-25T17:34:28.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.096+0000 7f040affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:28.097 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.096+0000 7f040affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:28.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.340+0000 7f041c3ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:28.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.340+0000 7f041c3ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:28.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.340+0000 7f041c3ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:28.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.340+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0414141a00 con 0x7f04140a98f0 2024-03-25T17:34:28.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.373+0000 7f040affd640 1 -- 172.21.15.134:0/3298116502 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f04100522b0 con 0x7f04140a98f0 2024-03-25T17:34:28.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.373+0000 7f040affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:28.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.373+0000 7f040affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:28.375 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:28.375 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 10 librados: watch_flush enter 2024-03-25T17:34:28.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 10 librados: watch_flush exit 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 10 client.4456.objecter successfully canceled tick 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 >> v1:172.21.15.134:6800/610837766 conn(0x7f03f806dd70 legacy=0x7f03f8070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 10 monclient: shutdown 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 >> v1:172.21.15.159:6789/0 conn(0x7f04140a98f0 legacy=0x7f0414137ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 shutdown_connections 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 >> 172.21.15.134:0/3298116502 conn(0x7f04140b7ea0 msgr2=0x7f0414074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 shutdown_connections 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 1 -- 172.21.15.134:0/3298116502 wait complete. 2024-03-25T17:34:28.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:28.376+0000 7f041c3ba640 1 librados: shutdown 2024-03-25T17:34:28.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:29.873 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:34.889 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.890 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:34.906 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.914 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:34.930 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.917 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:34.946 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.936 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:34.952 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.939 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:34.956 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.942 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:34.971 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.965 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:34.994 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:29.970 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:34.986 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:35.194 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:35.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.310+0000 7f0b687d6640 10 monclient: get_monmap_and_config 2024-03-25T17:34:35.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.310+0000 7f0b687d6640 10 monclient: build_initial_monmap 2024-03-25T17:34:35.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.310+0000 7f0b687d6640 10 monclient: monmap: 2024-03-25T17:34:35.311 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:35.311 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:35.311170+0000 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:35.311170+0000 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.310+0000 7f0b687d6640 1 Processor -- start 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.311+0000 7f0b687d6640 1 -- start start 2024-03-25T17:34:35.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.311+0000 7f0b687d6640 10 monclient: init 2024-03-25T17:34:35.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.311+0000 7f0b687d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:35.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:34:35.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b600b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:35.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b600b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:35.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b6014be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:35.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600af940 con 0x7f0b600b4d60 2024-03-25T17:34:35.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600b0a70 con 0x7f0b600b1730 2024-03-25T17:34:35.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b600b0d50 con 0x7f0b6014be90 2024-03-25T17:34:35.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:35.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.312+0000 7f0b687d6640 10 monclient(hunting): authenticate will time out at 5628.574958s 2024-03-25T17:34:35.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.320+0000 7f0b65d71640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f0b600b1730 0x7f0b600b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:36254/0 (socket says 172.21.15.134:36254) 2024-03-25T17:34:35.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.320+0000 7f0b65d71640 1 -- 172.21.15.134:0/3864883885 learned_addr learned my addr 172.21.15.134:0/3864883885 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:35.334 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.334+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028140797 0 0) 0x7f0b600b0a70 con 0x7f0b600b1730 2024-03-25T17:34:35.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.334+0000 7f0b65570640 10 monclient(hunting): _negotiate 2024-03-25T17:34:35.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.334+0000 7f0b65570640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:35.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.334+0000 7f0b65570640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:35.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.334+0000 7f0b65570640 10 monclient(hunting): my global_id is 4459 2024-03-25T17:34:35.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.334+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b4c003520 con 0x7f0b600b1730 2024-03-25T17:34:35.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.342+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919204832 0 0) 0x7f0b600b0d50 con 0x7f0b6014be90 2024-03-25T17:34:35.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.342+0000 7f0b65570640 10 monclient(hunting): _negotiate 2024-03-25T17:34:35.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.342+0000 7f0b65570640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:35.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.342+0000 7f0b65570640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:35.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.342+0000 7f0b65570640 10 monclient(hunting): my global_id is 34424 2024-03-25T17:34:35.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.342+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b600b0a70 con 0x7f0b6014be90 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3195668240 0 0) 0x7f0b4c003520 con 0x7f0b600b1730 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 >> v1:172.21.15.134:6790/0 conn(0x7f0b6014be90 legacy=0x7f0b6014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 >> v1:172.21.15.134:6789/0 conn(0x7f0b600b4d60 legacy=0x7f0b6013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 10 monclient: _finish_hunting 0 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 1 monclient: found mon.noname-c 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b600b1030 con 0x7f0b600b1730 2024-03-25T17:34:35.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 10 monclient: _finish_auth 0 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 10 monclient: _check_auth_tickets 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0b50001690 con 0x7f0b600b1730 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b65570640 10 monclient: dump: 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:35.349 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:35.350 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:35.350 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:35.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b687d6640 5 monclient: authenticate success, global_id 4459 2024-03-25T17:34:35.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.347+0000 7f0b687d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:35.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b500017f0 con 0x7f0b600b1730 2024-03-25T17:34:35.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b65570640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:35.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b65570640 1 -- 172.21.15.134:0/3864883885 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0b50001c00 con 0x7f0b600b1730 2024-03-25T17:34:35.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b65570640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:35.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b65570640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:35.353 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b65570640 10 monclient: dump: 2024-03-25T17:34:35.353 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b687d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:35.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.352+0000 7f0b687d6640 4 set_mon_vals no callback set 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.353+0000 7f0b687d6640 10 monclient: shutdown 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.353+0000 7f0b687d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.353+0000 7f0b687d6640 1 -- 172.21.15.134:0/3864883885 >> v1:172.21.15.159:6789/0 conn(0x7f0b600b1730 legacy=0x7f0b600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.353+0000 7f0b687d6640 1 -- 172.21.15.134:0/3864883885 shutdown_connections 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.353+0000 7f0b687d6640 1 -- 172.21.15.134:0/3864883885 >> 172.21.15.134:0/3864883885 conn(0x7f0b600617a0 msgr2=0x7f0b60061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.353+0000 7f0b687d6640 1 -- 172.21.15.134:0/3864883885 shutdown_connections 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.353+0000 7f0b687d6640 1 -- 172.21.15.134:0/3864883885 wait complete. 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.354+0000 7f0b687d6640 10 monclient: build_initial_monmap 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.354+0000 7f0b687d6640 10 monclient: monmap: 2024-03-25T17:34:35.356 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.355+0000 7f0b687d6640 1 librados: starting msgr at 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.355+0000 7f0b687d6640 1 librados: starting objecter 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.355+0000 7f0b687d6640 1 Processor -- start 2024-03-25T17:34:35.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.355+0000 7f0b687d6640 1 -- start start 2024-03-25T17:34:35.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.355+0000 7f0b687d6640 1 librados: setting wanted keys 2024-03-25T17:34:35.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.355+0000 7f0b687d6640 1 librados: calling monclient init 2024-03-25T17:34:35.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.355+0000 7f0b687d6640 10 monclient: init 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0b600b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0b600b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0b6014be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b60142050 con 0x7f0b6014be90 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b601dac50 con 0x7f0b600b1730 2024-03-25T17:34:35.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b60140d00 con 0x7f0b600b4d60 2024-03-25T17:34:35.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:35.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.356+0000 7f0b687d6640 10 monclient(hunting): authenticate will time out at 5628.619116s 2024-03-25T17:34:35.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.371+0000 7f0b66d73640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f0b6014be90 0x7f0b601e2260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49746/0 (socket says 172.21.15.134:49746) 2024-03-25T17:34:35.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.371+0000 7f0b66d73640 1 -- 172.21.15.134:0/4090958026 learned_addr learned my addr 172.21.15.134:0/4090958026 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:35.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.374+0000 7f0b56ffd640 10 client.?.objecter ms_handle_connect 0x7f0b600b1730 2024-03-25T17:34:35.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.374+0000 7f0b56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.378+0000 7f0b56ffd640 10 client.?.objecter ms_handle_connect 0x7f0b6014be90 2024-03-25T17:34:35.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.378+0000 7f0b56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:35.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.383+0000 7f0b56ffd640 10 client.?.objecter ms_handle_connect 0x7f0b600b4d60 2024-03-25T17:34:35.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.383+0000 7f0b56ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:35.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.391+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322785922 0 0) 0x7f0b60142050 con 0x7f0b6014be90 2024-03-25T17:34:35.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.391+0000 7f0b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:35.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.391+0000 7f0b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:35.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.391+0000 7f0b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:35.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.391+0000 7f0b56ffd640 10 monclient(hunting): my global_id is 4440 2024-03-25T17:34:35.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.391+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b44004eb0 con 0x7f0b6014be90 2024-03-25T17:34:35.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.400+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624472232 0 0) 0x7f0b60140d00 con 0x7f0b600b4d60 2024-03-25T17:34:35.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.400+0000 7f0b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:35.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.400+0000 7f0b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:35.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.400+0000 7f0b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:35.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.400+0000 7f0b56ffd640 10 monclient(hunting): my global_id is 34427 2024-03-25T17:34:35.401 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.400+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b60142050 con 0x7f0b600b4d60 2024-03-25T17:34:35.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.404+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387316052 0 0) 0x7f0b44004eb0 con 0x7f0b6014be90 2024-03-25T17:34:35.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.404+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b60140d00 con 0x7f0b6014be90 2024-03-25T17:34:35.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.404+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0b58001c90 con 0x7f0b6014be90 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.404+0000 7f0b56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.404+0000 7f0b56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.404+0000 7f0b56ffd640 10 monclient(hunting): dump: 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:35.405 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:35.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.404+0000 7f0b56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:35.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.407+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316318559 0 0) 0x7f0b601dac50 con 0x7f0b600b1730 2024-03-25T17:34:35.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.407+0000 7f0b56ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:35.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.407+0000 7f0b56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:35.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.407+0000 7f0b56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:35.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.407+0000 7f0b56ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T17:34:35.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.407+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b44004eb0 con 0x7f0b600b1730 2024-03-25T17:34:35.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2129100811 0 0) 0x7f0b60140d00 con 0x7f0b6014be90 2024-03-25T17:34:35.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 >> v1:172.21.15.134:6790/0 conn(0x7f0b600b4d60 legacy=0x7f0b601db150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:35.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 >> v1:172.21.15.159:6789/0 conn(0x7f0b600b1730 legacy=0x7f0b601405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 1 monclient: found mon.a 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b601423e0 con 0x7f0b6014be90 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 10 monclient: _finish_auth 0 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 5 monclient: authenticate success, global_id 4440 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 10 monclient: _renew_subs 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6013b470 con 0x7f0b6014be90 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:35.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 10 monclient: _renew_subs 2024-03-25T17:34:35.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:35.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b601dba40 con 0x7f0b6014be90 2024-03-25T17:34:35.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.421+0000 7f0b687d6640 1 librados: init done 2024-03-25T17:34:35.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.423+0000 7f0b687d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:35.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.423+0000 7f0b687d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:35.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.423+0000 7f0b687d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:35.425 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.423+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b601401f0 con 0x7f0b6014be90 2024-03-25T17:34:35.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.731+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b58003990 con 0x7f0b6014be90 2024-03-25T17:34:35.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.731+0000 7f0b56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:35.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.731+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f0b58003d60 con 0x7f0b6014be90 2024-03-25T17:34:35.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.731+0000 7f0b56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:35.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.731+0000 7f0b56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:35.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.731+0000 7f0b56ffd640 10 monclient: dump: 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:35.733 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.731+0000 7f0b64d6f640 4 set_mon_vals no callback set 2024-03-25T17:34:35.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.732+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f0b58003f80 con 0x7f0b6014be90 2024-03-25T17:34:35.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.733+0000 7f0b66572640 10 monclient: get_auth_request con 0x7f0b4406de20 auth_method 0 2024-03-25T17:34:35.734 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.733+0000 7f0b56ffd640 10 client.4440.objecter ms_handle_connect 0x7f0b4406de20 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f0b58054a20 con 0x7f0b6014be90 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 10 client.4440.objecter ms_dispatch 0x7f0b60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b58050530 con 0x7f0b6014be90 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:35.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.737+0000 7f0b56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:35.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.979+0000 7f0b687d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:35.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.979+0000 7f0b687d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:35.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.979+0000 7f0b687d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:35.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:35.979+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b60140e90 con 0x7f0b6014be90 2024-03-25T17:34:36.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.005+0000 7f0b56ffd640 1 -- 172.21.15.134:0/4090958026 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f0b58052460 con 0x7f0b6014be90 2024-03-25T17:34:36.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.005+0000 7f0b56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:36.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.005+0000 7f0b56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:36.006 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:36.006 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:36.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.007+0000 7f0b687d6640 10 librados: watch_flush enter 2024-03-25T17:34:36.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.007+0000 7f0b687d6640 10 librados: watch_flush exit 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.007+0000 7f0b687d6640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.007+0000 7f0b687d6640 10 client.4440.objecter successfully canceled tick 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 >> v1:172.21.15.134:6800/610837766 conn(0x7f0b4406de20 legacy=0x7f0b440702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 10 monclient: shutdown 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 >> v1:172.21.15.134:6789/0 conn(0x7f0b6014be90 legacy=0x7f0b601e2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 shutdown_connections 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 >> 172.21.15.134:0/4090958026 conn(0x7f0b600617a0 msgr2=0x7f0b600b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 shutdown_connections 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 1 -- 172.21.15.134:0/4090958026 wait complete. 2024-03-25T17:34:36.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:36.008+0000 7f0b687d6640 1 librados: shutdown 2024-03-25T17:34:36.020 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:42.827 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:42.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.940+0000 7f515ea63640 10 monclient: get_monmap_and_config 2024-03-25T17:34:42.941 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.940+0000 7f515ea63640 10 monclient: build_initial_monmap 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.940+0000 7f515ea63640 10 monclient: monmap: 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:42.941580+0000 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:42.941580+0000 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:42.942 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:42.943 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:42.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.941+0000 7f515ea63640 1 Processor -- start 2024-03-25T17:34:42.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.941+0000 7f515ea63640 1 -- start start 2024-03-25T17:34:42.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.941+0000 7f515ea63640 10 monclient: init 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.942+0000 7f515ea63640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.942+0000 7f515ea63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.942+0000 7f515ea63640 10 monclient(hunting): picked mon.noname-b con 0x7f51580a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.942+0000 7f515ea63640 10 monclient(hunting): picked mon.noname-a con 0x7f515813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.942+0000 7f515ea63640 10 monclient(hunting): picked mon.noname-c con 0x7f51581430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.942+0000 7f515ea63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5158142520 con 0x7f515813f710 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.942+0000 7f515ea63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580b28a0 con 0x7f51581430c0 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.943+0000 7f515ea63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580b2a30 con 0x7f51580a98f0 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.943+0000 7f515ea63640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:42.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.943+0000 7f515ea63640 10 monclient(hunting): authenticate will time out at 5636.205795s 2024-03-25T17:34:42.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.949+0000 7f515d000640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f51581430c0 0x7f515814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:57032/0 (socket says 172.21.15.134:57032) 2024-03-25T17:34:42.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.949+0000 7f515d000640 1 -- 172.21.15.134:0/2175792447 learned_addr learned my addr 172.21.15.134:0/2175792447 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:42.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.968+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041363858 0 0) 0x7f5158142520 con 0x7f515813f710 2024-03-25T17:34:42.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.968+0000 7f5156ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:42.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.968+0000 7f5156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:42.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.969+0000 7f5156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:42.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.969+0000 7f5156ffd640 10 monclient(hunting): my global_id is 4443 2024-03-25T17:34:42.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.969+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5140003520 con 0x7f515813f710 2024-03-25T17:34:42.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.979+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879897069 0 0) 0x7f51580b28a0 con 0x7f51581430c0 2024-03-25T17:34:43.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.979+0000 7f5156ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:43.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.979+0000 7f5156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:43.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.979+0000 7f5156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:43.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.979+0000 7f5156ffd640 10 monclient(hunting): my global_id is 4465 2024-03-25T17:34:43.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.979+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5158142520 con 0x7f51581430c0 2024-03-25T17:34:43.026 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.982+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667497366 0 0) 0x7f51580b2a30 con 0x7f51580a98f0 2024-03-25T17:34:43.026 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.982+0000 7f5156ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:43.026 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.982+0000 7f5156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.982+0000 7f5156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.982+0000 7f5156ffd640 10 monclient(hunting): my global_id is 34430 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.982+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51580b28a0 con 0x7f51580a98f0 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3501093041 0 0) 0x7f5140003520 con 0x7f515813f710 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 >> v1:172.21.15.134:6790/0 conn(0x7f51580a98f0 legacy=0x7f51580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 >> v1:172.21.15.159:6789/0 conn(0x7f51581430c0 legacy=0x7f515814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 1 monclient: found mon.noname-a 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:34:43.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580b44c0 con 0x7f515813f710 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 10 monclient: _finish_auth 0 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f514c001690 con 0x7f515813f710 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.985+0000 7f515ea63640 5 monclient: authenticate success, global_id 4443 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.986+0000 7f515ea63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.986+0000 7f5156ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.986+0000 7f5156ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.986+0000 7f5156ffd640 10 monclient: dump: 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:43.028 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:42.986+0000 7f515ea63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f514c0017f0 con 0x7f515813f710 2024-03-25T17:34:43.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f5156ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f5156ffd640 1 -- 172.21.15.134:0/2175792447 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f514c001c00 con 0x7f515813f710 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 4 set_mon_vals no callback set 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f5156ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f5156ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f5156ffd640 10 monclient: dump: 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:43.030 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 10 monclient: shutdown 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 1 -- 172.21.15.134:0/2175792447 >> v1:172.21.15.134:6789/0 conn(0x7f515813f710 legacy=0x7f51581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 1 -- 172.21.15.134:0/2175792447 shutdown_connections 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 1 -- 172.21.15.134:0/2175792447 >> 172.21.15.134:0/2175792447 conn(0x7f51580b7ea0 msgr2=0x7f51580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 1 -- 172.21.15.134:0/2175792447 shutdown_connections 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.013+0000 7f515ea63640 1 -- 172.21.15.134:0/2175792447 wait complete. 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.014+0000 7f515ea63640 10 monclient: build_initial_monmap 2024-03-25T17:34:43.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.014+0000 7f515ea63640 10 monclient: monmap: 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.015+0000 7f515ea63640 1 librados: starting msgr at 2024-03-25T17:34:43.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.015+0000 7f515ea63640 1 librados: starting objecter 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.015+0000 7f515ea63640 1 Processor -- start 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.015+0000 7f515ea63640 1 -- start start 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.015+0000 7f515ea63640 1 librados: setting wanted keys 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.015+0000 7f515ea63640 1 librados: calling monclient init 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.015+0000 7f515ea63640 10 monclient: init 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 10 monclient(hunting): picked mon.noname-b con 0x7f51580a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 10 monclient(hunting): picked mon.noname-c con 0x7f515813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 10 monclient(hunting): picked mon.noname-a con 0x7f51581430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5158135070 con 0x7f51581430c0 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580b75c0 con 0x7f51580a98f0 2024-03-25T17:34:43.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51580b7750 con 0x7f515813f710 2024-03-25T17:34:43.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:43.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.016+0000 7f515ea63640 10 monclient(hunting): authenticate will time out at 5636.279082s 2024-03-25T17:34:43.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.022+0000 7f515d000640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f51581430c0 0x7f51581e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:48078/0 (socket says 172.21.15.134:48078) 2024-03-25T17:34:43.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.022+0000 7f515d000640 1 -- 172.21.15.134:0/1456291972 learned_addr learned my addr 172.21.15.134:0/1456291972 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:43.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.023+0000 7f5154ff9640 10 client.?.objecter ms_handle_connect 0x7f51581430c0 2024-03-25T17:34:43.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.023+0000 7f5154ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:43.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.035+0000 7f5154ff9640 10 client.?.objecter ms_handle_connect 0x7f51580a98f0 2024-03-25T17:34:43.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.035+0000 7f5154ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:43.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.037+0000 7f5154ff9640 10 client.?.objecter ms_handle_connect 0x7f515813f710 2024-03-25T17:34:43.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.037+0000 7f5154ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:43.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.040+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548765527 0 0) 0x7f5158135070 con 0x7f51581430c0 2024-03-25T17:34:43.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.040+0000 7f5154ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:43.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.040+0000 7f5154ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:43.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.040+0000 7f5154ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:43.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.040+0000 7f5154ff9640 10 monclient(hunting): my global_id is 4446 2024-03-25T17:34:43.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.040+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5144004eb0 con 0x7f51581430c0 2024-03-25T17:34:43.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.046+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441994378 0 0) 0x7f51580b7750 con 0x7f515813f710 2024-03-25T17:34:43.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.046+0000 7f5154ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:43.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.046+0000 7f5154ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:43.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.046+0000 7f5154ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:43.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.046+0000 7f5154ff9640 10 monclient(hunting): my global_id is 34433 2024-03-25T17:34:43.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.046+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5158135070 con 0x7f515813f710 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.053+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1768127708 0 0) 0x7f5144004eb0 con 0x7f51581430c0 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.053+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51580b7750 con 0x7f51581430c0 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.053+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5150001c70 con 0x7f51581430c0 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.053+0000 7f5154ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.053+0000 7f5154ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.053+0000 7f5154ff9640 10 monclient(hunting): dump: 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:43.054 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.053+0000 7f5154ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:43.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.054+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425649235 0 0) 0x7f51580b75c0 con 0x7f51580a98f0 2024-03-25T17:34:43.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.054+0000 7f5154ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:34:43.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.054+0000 7f5154ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:43.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.054+0000 7f5154ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:43.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.054+0000 7f5154ff9640 10 monclient(hunting): my global_id is 4468 2024-03-25T17:34:43.056 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.054+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5144004eb0 con 0x7f51580a98f0 2024-03-25T17:34:43.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 269651195 0 0) 0x7f51580b7750 con 0x7f51581430c0 2024-03-25T17:34:43.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 >> v1:172.21.15.134:6790/0 conn(0x7f515813f710 legacy=0x7f51581d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 >> v1:172.21.15.159:6789/0 conn(0x7f51580a98f0 legacy=0x7f51581d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 1 monclient: found mon.a 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5158135300 con 0x7f51581430c0 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 10 monclient: _finish_auth 0 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f5154ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 5 monclient: authenticate success, global_id 4446 2024-03-25T17:34:43.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 10 monclient: _renew_subs 2024-03-25T17:34:43.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:43.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5158135550 con 0x7f51581430c0 2024-03-25T17:34:43.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:43.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 10 monclient: _renew_subs 2024-03-25T17:34:43.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:43.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f515813b620 con 0x7f51581430c0 2024-03-25T17:34:43.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.063+0000 7f515ea63640 1 librados: init done 2024-03-25T17:34:43.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.064+0000 7f515ea63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:43.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.064+0000 7f515ea63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:43.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.064+0000 7f515ea63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:43.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.064+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51580627b0 con 0x7f51581430c0 2024-03-25T17:34:43.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5150004b60 con 0x7f51581430c0 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f5154ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5150004e70 con 0x7f51581430c0 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f5154ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f5154ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f5154ff9640 10 monclient: dump: 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.365 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:43.366 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:43.366 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:43.366 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:43.366 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:43.366 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:43.366 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:43.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f51567fc640 4 set_mon_vals no callback set 2024-03-25T17:34:43.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.364+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f515001ae40 con 0x7f51581430c0 2024-03-25T17:34:43.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.365+0000 7f5157fff640 10 monclient: get_auth_request con 0x7f514406de00 auth_method 0 2024-03-25T17:34:43.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.365+0000 7f5154ff9640 10 client.4446.objecter ms_handle_connect 0x7f514406de00 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f5150054b00 con 0x7f51581430c0 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 10 client.4446.objecter ms_dispatch 0x7f5158000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5150050440 con 0x7f51581430c0 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:43.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.371+0000 7f5154ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:43.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.621+0000 7f515ea63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:43.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.621+0000 7f515ea63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:43.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.621+0000 7f515ea63640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:34:43.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.621+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51580b7b10 con 0x7f51581430c0 2024-03-25T17:34:43.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.697+0000 7f5154ff9640 1 -- 172.21.15.134:0/1456291972 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f5150052540 con 0x7f51581430c0 2024-03-25T17:34:43.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.697+0000 7f5154ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:43.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.697+0000 7f5154ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:43.698 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:43.699 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:43.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 10 librados: watch_flush enter 2024-03-25T17:34:43.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 10 librados: watch_flush exit 2024-03-25T17:34:43.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-25T17:34:43.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 10 client.4446.objecter successfully canceled tick 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 >> v1:172.21.15.134:6800/610837766 conn(0x7f514406de00 legacy=0x7f51440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 10 monclient: shutdown 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 >> v1:172.21.15.134:6789/0 conn(0x7f51581430c0 legacy=0x7f51581e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 shutdown_connections 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.700+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 >> 172.21.15.134:0/1456291972 conn(0x7f51580b7ea0 msgr2=0x7f5158074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.701+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 shutdown_connections 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.701+0000 7f515ea63640 1 -- 172.21.15.134:0/1456291972 wait complete. 2024-03-25T17:34:43.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:43.701+0000 7f515ea63640 1 librados: shutdown 2024-03-25T17:34:43.713 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:49.919 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:54.935 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:49.927 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:54.943 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:49.959 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:05:54.975 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:49.975 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:55.003 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:49.975 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:54.991 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:49.985 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:55.014 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:49.988 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:55.004 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:50.005 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:05:55.034 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:50.023 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:05:55.039 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:34:50.520 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:50.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.631+0000 7f19b17cd640 10 monclient: get_monmap_and_config 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.631+0000 7f19b17cd640 10 monclient: build_initial_monmap 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.631+0000 7f19b17cd640 10 monclient: monmap: 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:50.632943+0000 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:50.632943+0000 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:50.633 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.632+0000 7f19b17cd640 1 Processor -- start 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.632+0000 7f19b17cd640 1 -- start start 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.632+0000 7f19b17cd640 10 monclient: init 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 10 monclient(hunting): picked mon.noname-a con 0x7f19ac0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 10 monclient(hunting): picked mon.noname-b con 0x7f19ac13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 10 monclient(hunting): picked mon.noname-c con 0x7f19ac1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:50.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac142520 con 0x7f19ac0a98f0 2024-03-25T17:34:50.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac0b28a0 con 0x7f19ac1430c0 2024-03-25T17:34:50.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac0b2a30 con 0x7f19ac13f710 2024-03-25T17:34:50.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:50.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.633+0000 7f19b17cd640 10 monclient(hunting): authenticate will time out at 5643.896275s 2024-03-25T17:34:50.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.639+0000 7f19aaffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f19ac0a98f0 0x7f19ac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39530/0 (socket says 172.21.15.134:39530) 2024-03-25T17:34:50.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.639+0000 7f19aaffd640 1 -- 172.21.15.134:0/3323075106 learned_addr learned my addr 172.21.15.134:0/3323075106 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:50.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.646+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121615330 0 0) 0x7f19ac0b2a30 con 0x7f19ac13f710 2024-03-25T17:34:50.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.646+0000 7f19aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:50.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.646+0000 7f19aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:50.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.646+0000 7f19aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:50.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.646+0000 7f19aa7fc640 10 monclient(hunting): my global_id is 34436 2024-03-25T17:34:50.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.646+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1998003520 con 0x7f19ac13f710 2024-03-25T17:34:50.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.658+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60063142 0 0) 0x7f19ac142520 con 0x7f19ac0a98f0 2024-03-25T17:34:50.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.658+0000 7f19aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:34:50.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.658+0000 7f19aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:50.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.658+0000 7f19aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:50.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.658+0000 7f19aa7fc640 10 monclient(hunting): my global_id is 4449 2024-03-25T17:34:50.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.658+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac0b2a30 con 0x7f19ac0a98f0 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 528078983 0 0) 0x7f1998003520 con 0x7f19ac13f710 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 >> v1:172.21.15.159:6789/0 conn(0x7f19ac1430c0 legacy=0x7f19ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 >> v1:172.21.15.134:6789/0 conn(0x7f19ac0a98f0 legacy=0x7f19ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 1 monclient: found mon.noname-b 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:34:50.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac0b44c0 con 0x7f19ac13f710 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 10 monclient: _finish_auth 0 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1994001690 con 0x7f19ac13f710 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19aa7fc640 10 monclient: dump: 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:50.663 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:50.664 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:50.664 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:50.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19b17cd640 5 monclient: authenticate success, global_id 34436 2024-03-25T17:34:50.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.661+0000 7f19b17cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:50.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19940017f0 con 0x7f19ac13f710 2024-03-25T17:34:50.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:50.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19aa7fc640 1 -- 172.21.15.134:0/3323075106 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1994001c00 con 0x7f19ac13f710 2024-03-25T17:34:50.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19b17cd640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:50.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19b17cd640 4 set_mon_vals no callback set 2024-03-25T17:34:50.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:50.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.666+0000 7f19aa7fc640 10 monclient: dump: 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:50.668 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:50.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.667+0000 7f19b17cd640 10 monclient: shutdown 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.667+0000 7f19b17cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.667+0000 7f19b17cd640 1 -- 172.21.15.134:0/3323075106 >> v1:172.21.15.134:6790/0 conn(0x7f19ac13f710 legacy=0x7f19ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.667+0000 7f19b17cd640 1 -- 172.21.15.134:0/3323075106 shutdown_connections 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.667+0000 7f19b17cd640 1 -- 172.21.15.134:0/3323075106 >> 172.21.15.134:0/3323075106 conn(0x7f19ac0b7ea0 msgr2=0x7f19ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.667+0000 7f19b17cd640 1 -- 172.21.15.134:0/3323075106 shutdown_connections 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.667+0000 7f19b17cd640 1 -- 172.21.15.134:0/3323075106 wait complete. 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 10 monclient: build_initial_monmap 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 10 monclient: monmap: 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:50.671 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 1 librados: starting msgr at 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 1 librados: starting objecter 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 1 Processor -- start 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 1 -- start start 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 1 librados: setting wanted keys 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 1 librados: calling monclient init 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.668+0000 7f19b17cd640 10 monclient: init 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:34:50.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 10 monclient(hunting): picked mon.noname-b con 0x7f19ac0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:50.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 10 monclient(hunting): picked mon.noname-a con 0x7f19ac13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:50.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 10 monclient(hunting): picked mon.noname-c con 0x7f19ac1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:50.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac0aba00 con 0x7f19ac13f710 2024-03-25T17:34:50.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac1d7490 con 0x7f19ac0a98f0 2024-03-25T17:34:50.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19ac1d8360 con 0x7f19ac1430c0 2024-03-25T17:34:50.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:50.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.669+0000 7f19b17cd640 10 monclient(hunting): authenticate will time out at 5643.932375s 2024-03-25T17:34:50.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.676+0000 7f19a3fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f19ac13f710 0x7f19ac1d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39546/0 (socket says 172.21.15.134:39546) 2024-03-25T17:34:50.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.676+0000 7f19a3fff640 1 -- 172.21.15.134:0/3417443151 learned_addr learned my addr 172.21.15.134:0/3417443151 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:50.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.678+0000 7f19a37fe640 10 client.?.objecter ms_handle_connect 0x7f19ac13f710 2024-03-25T17:34:50.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.678+0000 7f19a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:50.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.683+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300596563 0 0) 0x7f19ac0aba00 con 0x7f19ac13f710 2024-03-25T17:34:50.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.683+0000 7f19a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:34:50.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.683+0000 7f19a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:50.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.683+0000 7f19a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:50.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.683+0000 7f19a37fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T17:34:50.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.683+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f198c004eb0 con 0x7f19ac13f710 2024-03-25T17:34:50.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.687+0000 7f19a37fe640 10 client.?.objecter ms_handle_connect 0x7f19ac1430c0 2024-03-25T17:34:50.687 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.687+0000 7f19a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:50.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.688+0000 7f19a37fe640 10 client.?.objecter ms_handle_connect 0x7f19ac0a98f0 2024-03-25T17:34:50.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.688+0000 7f19a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:50.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.690+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730252810 0 0) 0x7f19ac1d8360 con 0x7f19ac1430c0 2024-03-25T17:34:50.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.690+0000 7f19a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:34:50.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.690+0000 7f19a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:50.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.690+0000 7f19a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:50.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.690+0000 7f19a37fe640 10 monclient(hunting): my global_id is 34439 2024-03-25T17:34:50.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.690+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac0aba00 con 0x7f19ac1430c0 2024-03-25T17:34:50.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.696+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149642736 0 0) 0x7f19ac1d7490 con 0x7f19ac0a98f0 2024-03-25T17:34:50.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.696+0000 7f19a37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:34:50.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.696+0000 7f19a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:50.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.696+0000 7f19a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:50.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.696+0000 7f19a37fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T17:34:50.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.696+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac1d8360 con 0x7f19ac0a98f0 2024-03-25T17:34:50.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.701+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3900575323 0 0) 0x7f19ac0aba00 con 0x7f19ac1430c0 2024-03-25T17:34:50.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.701+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19ac1d7490 con 0x7f19ac1430c0 2024-03-25T17:34:50.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.702+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f19a4001bb0 con 0x7f19ac1430c0 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.702+0000 7f19a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.702+0000 7f19a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.702+0000 7f19a37fe640 10 monclient(hunting): dump: 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:50.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.702+0000 7f19a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:50.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 147391035 0 0) 0x7f19ac1d7490 con 0x7f19ac1430c0 2024-03-25T17:34:50.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 >> v1:172.21.15.159:6789/0 conn(0x7f19ac0a98f0 legacy=0x7f19ac1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:50.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 >> v1:172.21.15.134:6789/0 conn(0x7f19ac13f710 legacy=0x7f19ac1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:50.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:34:50.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 1 monclient: found mon.c 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac0abc30 con 0x7f19ac1430c0 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 10 monclient: _finish_auth 0 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 5 monclient: authenticate success, global_id 34439 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 10 monclient: _renew_subs 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac137940 con 0x7f19ac1430c0 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 10 client.34439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 10 monclient: _renew_subs 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:34:50.709 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac135220 con 0x7f19ac1430c0 2024-03-25T17:34:50.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.707+0000 7f19b17cd640 1 librados: init done 2024-03-25T17:34:50.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.708+0000 7f19b17cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:50.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.708+0000 7f19b17cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:50.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.708+0000 7f19b17cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:34:50.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:50.708+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19ac0627b0 con 0x7f19ac1430c0 2024-03-25T17:34:51.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.013+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19a4002080 con 0x7f19ac1430c0 2024-03-25T17:34:51.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.013+0000 7f19a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:51.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.013+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f19a4003b30 con 0x7f19ac1430c0 2024-03-25T17:34:51.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.013+0000 7f19a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:51.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.013+0000 7f19a37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:34:51.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.013+0000 7f19a37fe640 10 monclient: dump: 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:51.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.013+0000 7f19a9ffb640 4 set_mon_vals no callback set 2024-03-25T17:34:51.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.014+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f19a4003d50 con 0x7f19ac1430c0 2024-03-25T17:34:51.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.015+0000 7f19aaffd640 10 monclient: get_auth_request con 0x7f198c06dda0 auth_method 0 2024-03-25T17:34:51.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.015+0000 7f19a37fe640 10 client.34439.objecter ms_handle_connect 0x7f198c06dda0 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f19a4053880 con 0x7f19ac1430c0 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 10 client.34439.objecter ms_dispatch 0x7f19ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 3 client.34439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 3 client.34439.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 20 client.34439.objecter dump_active .. 0 homeless 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19a404f1c0 con 0x7f19ac1430c0 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:51.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.022+0000 7f19a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:51.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.239+0000 7f19b17cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:51.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.239+0000 7f19b17cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:51.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.239+0000 7f19b17cd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:34:51.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.239+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19ac1d7830 con 0x7f19ac1430c0 2024-03-25T17:34:51.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.355+0000 7f19a37fe640 1 -- 172.21.15.134:0/3417443151 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f19a40512c0 con 0x7f19ac1430c0 2024-03-25T17:34:51.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.355+0000 7f19a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:51.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.355+0000 7f19a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:51.356 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:51.356 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 10 librados: watch_flush enter 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 10 librados: watch_flush exit 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 20 client.34439.objecter shutdown clearing up homeless session... 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 10 client.34439.objecter successfully canceled tick 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 >> v1:172.21.15.134:6800/610837766 conn(0x7f198c06dda0 legacy=0x7f198c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 10 monclient: shutdown 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:51.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 >> v1:172.21.15.134:6790/0 conn(0x7f19ac1430c0 legacy=0x7f19ac133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:51.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 shutdown_connections 2024-03-25T17:34:51.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 >> 172.21.15.134:0/3417443151 conn(0x7f19ac0b7ea0 msgr2=0x7f19ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:51.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 shutdown_connections 2024-03-25T17:34:51.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 1 -- 172.21.15.134:0/3417443151 wait complete. 2024-03-25T17:34:51.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:51.357+0000 7f19b17cd640 1 librados: shutdown 2024-03-25T17:34:51.371 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:34:58.177 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:34:58.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.290+0000 7f196c4f1640 10 monclient: get_monmap_and_config 2024-03-25T17:34:58.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.290+0000 7f196c4f1640 10 monclient: build_initial_monmap 2024-03-25T17:34:58.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.290+0000 7f196c4f1640 10 monclient: monmap: 2024-03-25T17:34:58.291 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:58.291 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:34:58.291732+0000 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:34:58.291732+0000 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:34:58.292 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.291+0000 7f196c4f1640 1 Processor -- start 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.291+0000 7f196c4f1640 1 -- start start 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.291+0000 7f196c4f1640 10 monclient: init 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f19640a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f196413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:58.294 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f19641430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:58.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1964142520 con 0x7f19640a98f0 2024-03-25T17:34:58.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640b28a0 con 0x7f196413f710 2024-03-25T17:34:58.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640b2a30 con 0x7f19641430c0 2024-03-25T17:34:58.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:58.295 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.292+0000 7f196c4f1640 10 monclient(hunting): authenticate will time out at 5651.555603s 2024-03-25T17:34:58.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.307+0000 7f1969a8c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f196413f710 0x7f19641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:60932/0 (socket says 172.21.15.134:60932) 2024-03-25T17:34:58.307 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.307+0000 7f1969a8c640 1 -- 172.21.15.134:0/4138406019 learned_addr learned my addr 172.21.15.134:0/4138406019 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:58.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.328+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220605766 0 0) 0x7f19640b28a0 con 0x7f196413f710 2024-03-25T17:34:58.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.328+0000 7f196928b640 10 monclient(hunting): _negotiate 2024-03-25T17:34:58.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.328+0000 7f196928b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:58.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.328+0000 7f196928b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:58.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.328+0000 7f196928b640 10 monclient(hunting): my global_id is 4477 2024-03-25T17:34:58.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.328+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1950003520 con 0x7f196413f710 2024-03-25T17:34:58.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.337+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418191602 0 0) 0x7f1964142520 con 0x7f19640a98f0 2024-03-25T17:34:58.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.337+0000 7f196928b640 10 monclient(hunting): _negotiate 2024-03-25T17:34:58.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.337+0000 7f196928b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:58.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.337+0000 7f196928b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:58.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.337+0000 7f196928b640 10 monclient(hunting): my global_id is 4455 2024-03-25T17:34:58.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.337+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19640b28a0 con 0x7f19640a98f0 2024-03-25T17:34:58.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.338+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191089212 0 0) 0x7f19640b2a30 con 0x7f19641430c0 2024-03-25T17:34:58.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.338+0000 7f196928b640 10 monclient(hunting): _negotiate 2024-03-25T17:34:58.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.338+0000 7f196928b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:58.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.338+0000 7f196928b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:58.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.338+0000 7f196928b640 10 monclient(hunting): my global_id is 34442 2024-03-25T17:34:58.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.338+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1964142520 con 0x7f19641430c0 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1204344465 0 0) 0x7f19640b28a0 con 0x7f19640a98f0 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 >> v1:172.21.15.134:6790/0 conn(0x7f19641430c0 legacy=0x7f196414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 >> v1:172.21.15.159:6789/0 conn(0x7f196413f710 legacy=0x7f19641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 10 monclient: _finish_hunting 0 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 1 monclient: found mon.noname-a 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640b44c0 con 0x7f19640a98f0 2024-03-25T17:34:58.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 10 monclient: _finish_auth 0 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 10 monclient: _check_auth_tickets 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1960001670 con 0x7f19640a98f0 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196928b640 10 monclient: dump: 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:58.350 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196c4f1640 5 monclient: authenticate success, global_id 4455 2024-03-25T17:34:58.351 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.347+0000 7f196c4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:34:58.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19600017d0 con 0x7f19640a98f0 2024-03-25T17:34:58.365 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196928b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196928b640 1 -- 172.21.15.134:0/4138406019 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1960001be0 con 0x7f19640a98f0 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196928b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196928b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196928b640 10 monclient: dump: 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.366 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:58.367 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:58.367 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:58.367 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:58.367 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:58.367 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:58.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196c4f1640 10 monclient: get_monmap_and_config success 2024-03-25T17:34:58.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.364+0000 7f196c4f1640 4 set_mon_vals no callback set 2024-03-25T17:34:58.369 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.365+0000 7f196c4f1640 10 monclient: shutdown 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.365+0000 7f196c4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.365+0000 7f196c4f1640 1 -- 172.21.15.134:0/4138406019 >> v1:172.21.15.134:6789/0 conn(0x7f19640a98f0 legacy=0x7f19640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.365+0000 7f196c4f1640 1 -- 172.21.15.134:0/4138406019 shutdown_connections 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.365+0000 7f196c4f1640 1 -- 172.21.15.134:0/4138406019 >> 172.21.15.134:0/4138406019 conn(0x7f19640b7ea0 msgr2=0x7f19640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.365+0000 7f196c4f1640 1 -- 172.21.15.134:0/4138406019 shutdown_connections 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.365+0000 7f196c4f1640 1 -- 172.21.15.134:0/4138406019 wait complete. 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 10 monclient: build_initial_monmap 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 10 monclient: monmap: 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:34:58.370 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 1 librados: starting msgr at 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 1 librados: starting objecter 2024-03-25T17:34:58.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 1 Processor -- start 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 1 -- start start 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 1 librados: setting wanted keys 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 1 librados: calling monclient init 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.366+0000 7f196c4f1640 10 monclient: init 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 10 monclient: _reopen_session rank -1 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f19640a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f196413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f19641430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:34:58.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1964138f10 con 0x7f19641430c0 2024-03-25T17:34:58.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1964139970 con 0x7f196413f710 2024-03-25T17:34:58.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19640abb50 con 0x7f19640a98f0 2024-03-25T17:34:58.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 10 monclient(hunting): _renew_subs 2024-03-25T17:34:58.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.367+0000 7f196c4f1640 10 monclient(hunting): authenticate will time out at 5651.630383s 2024-03-25T17:34:58.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.382+0000 7f1969a8c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f196413f710 0x7f19640ac250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:32782/0 (socket says 172.21.15.134:32782) 2024-03-25T17:34:58.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.382+0000 7f1969a8c640 1 -- 172.21.15.134:0/2825051723 learned_addr learned my addr 172.21.15.134:0/2825051723 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:34:58.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.384+0000 7f195affd640 10 client.?.objecter ms_handle_connect 0x7f196413f710 2024-03-25T17:34:58.384 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.384+0000 7f195affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:58.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.386+0000 7f195affd640 10 client.?.objecter ms_handle_connect 0x7f19641430c0 2024-03-25T17:34:58.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.386+0000 7f195affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:58.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.386+0000 7f195affd640 10 client.?.objecter ms_handle_connect 0x7f19640a98f0 2024-03-25T17:34:58.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.386+0000 7f195affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:34:58.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.392+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125366303 0 0) 0x7f1964139970 con 0x7f196413f710 2024-03-25T17:34:58.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.392+0000 7f195affd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:58.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.392+0000 7f195affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:58.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.392+0000 7f195affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:58.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.392+0000 7f195affd640 10 monclient(hunting): my global_id is 4480 2024-03-25T17:34:58.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.392+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1948004eb0 con 0x7f196413f710 2024-03-25T17:34:58.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.394+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630019377 0 0) 0x7f19640abb50 con 0x7f19640a98f0 2024-03-25T17:34:58.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.394+0000 7f195affd640 10 monclient(hunting): _negotiate 2024-03-25T17:34:58.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.394+0000 7f195affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:34:58.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.394+0000 7f195affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:34:58.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.394+0000 7f195affd640 10 monclient(hunting): my global_id is 34445 2024-03-25T17:34:58.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.394+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1964139970 con 0x7f19640a98f0 2024-03-25T17:34:58.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.401+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1118651501 0 0) 0x7f1948004eb0 con 0x7f196413f710 2024-03-25T17:34:58.402 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.402+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19640abb50 con 0x7f196413f710 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.402+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1954001950 con 0x7f196413f710 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.402+0000 7f195affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.402+0000 7f195affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.402+0000 7f195affd640 10 monclient(hunting): dump: 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:58.403 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:58.404 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:58.404 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:58.404 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:58.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.402+0000 7f195affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:34:58.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 334192641 0 0) 0x7f19640abb50 con 0x7f196413f710 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 >> v1:172.21.15.134:6790/0 conn(0x7f19640a98f0 legacy=0x7f1964136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 >> v1:172.21.15.134:6789/0 conn(0x7f19641430c0 legacy=0x7f1964138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 10 monclient: _finish_hunting 0 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 1 monclient: found mon.b 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19640abe30 con 0x7f196413f710 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 10 monclient: _finish_auth 0 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 10 monclient: _check_auth_tickets 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f195affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:34:58.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 5 monclient: authenticate success, global_id 4480 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 10 monclient: _renew_subs 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1964137b40 con 0x7f196413f710 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 10 client.4480.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 10 monclient: _renew_subs 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1964138c10 con 0x7f196413f710 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.409+0000 7f196c4f1640 1 librados: init done 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.410+0000 7f196c4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.410+0000 7f196c4f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.410+0000 7f196c4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:58.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.410+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19640627b0 con 0x7f196413f710 2024-03-25T17:34:58.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.415+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f19540047e0 con 0x7f196413f710 2024-03-25T17:34:58.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.415+0000 7f195affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:34:58.416 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.415+0000 7f1968a8a640 4 set_mon_vals no callback set 2024-03-25T17:34:58.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.689+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1954001d80 con 0x7f196413f710 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.689+0000 7f195affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.689+0000 7f195affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.689+0000 7f195affd640 10 monclient: dump: 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:34:58.690 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:34:58.691 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.690+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f1954004bf0 con 0x7f196413f710 2024-03-25T17:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.690+0000 7f196a28d640 10 monclient: get_auth_request con 0x7f194806dc80 auth_method 0 2024-03-25T17:34:58.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.690+0000 7f195affd640 10 client.4480.objecter ms_handle_connect 0x7f194806dc80 2024-03-25T17:34:58.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.692+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f19540548a0 con 0x7f196413f710 2024-03-25T17:34:58.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.692+0000 7f195affd640 10 client.4480.objecter ms_dispatch 0x7f1964000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:34:58.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.692+0000 7f195affd640 3 client.4480.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:34:58.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.692+0000 7f195affd640 3 client.4480.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:34:58.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.692+0000 7f195affd640 20 client.4480.objecter dump_active .. 0 homeless 2024-03-25T17:34:58.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.696+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f19540501e0 con 0x7f196413f710 2024-03-25T17:34:58.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.696+0000 7f195affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:34:58.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.696+0000 7f195affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:34:58.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.921+0000 7f196c4f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:58.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.921+0000 7f196c4f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:58.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.921+0000 7f196c4f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:34:58.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.921+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19641390c0 con 0x7f196413f710 2024-03-25T17:34:58.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.939+0000 7f195affd640 1 -- 172.21.15.134:0/2825051723 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f19641390c0 con 0x7f196413f710 2024-03-25T17:34:58.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.939+0000 7f195affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:34:58.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.939+0000 7f195affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:34:58.940 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:34:58.940 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:34:58.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 10 librados: watch_flush enter 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 10 librados: watch_flush exit 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 20 client.4480.objecter shutdown clearing up homeless session... 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 10 client.4480.objecter successfully canceled tick 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 >> v1:172.21.15.134:6800/610837766 conn(0x7f194806dc80 legacy=0x7f1948070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 10 monclient: shutdown 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 >> v1:172.21.15.159:6789/0 conn(0x7f196413f710 legacy=0x7f19640ac250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 shutdown_connections 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.941+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 >> 172.21.15.134:0/2825051723 conn(0x7f19640b7ea0 msgr2=0x7f1964074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.942+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 shutdown_connections 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.942+0000 7f196c4f1640 1 -- 172.21.15.134:0/2825051723 wait complete. 2024-03-25T17:34:58.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:34:58.942+0000 7f196c4f1640 1 librados: shutdown 2024-03-25T17:34:58.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:05.761 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:05.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.880+0000 7fb0caa84640 10 monclient: get_monmap_and_config 2024-03-25T17:35:05.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.880+0000 7fb0caa84640 10 monclient: build_initial_monmap 2024-03-25T17:35:05.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.880+0000 7fb0caa84640 10 monclient: monmap: 2024-03-25T17:35:05.881 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:05.881 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:05.881 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:05.881783+0000 2024-03-25T17:35:05.881 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:05.881783+0000 2024-03-25T17:35:05.882 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:05.882 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:05.882 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:05.882 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:05.882 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:05.882 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.881+0000 7fb0caa84640 1 Processor -- start 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.881+0000 7fb0caa84640 1 -- start start 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.881+0000 7fb0caa84640 10 monclient: init 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 10 monclient(hunting): picked mon.noname-c con 0x7fb0c40a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 10 monclient(hunting): picked mon.noname-a con 0x7fb0c413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 10 monclient(hunting): picked mon.noname-b con 0x7fb0c41430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c4142520 con 0x7fb0c413f710 2024-03-25T17:35:05.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40b28a0 con 0x7fb0c40a98f0 2024-03-25T17:35:05.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c40b2a30 con 0x7fb0c41430c0 2024-03-25T17:35:05.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:05.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.882+0000 7fb0caa84640 10 monclient(hunting): authenticate will time out at 5659.145504s 2024-03-25T17:35:05.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.892+0000 7fb0c8820640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb0c40a98f0 0x7fb0c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58010/0 (socket says 172.21.15.134:58010) 2024-03-25T17:35:05.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.892+0000 7fb0c8820640 1 -- 172.21.15.134:0/2283057958 learned_addr learned my addr 172.21.15.134:0/2283057958 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220034623 0 0) 0x7fb0c40b28a0 con 0x7fb0c40a98f0 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): my global_id is 4483 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b4003520 con 0x7fb0c40a98f0 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949616207 0 0) 0x7fb0c4142520 con 0x7fb0c413f710 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 10 monclient(hunting): my global_id is 4461 2024-03-25T17:35:05.908 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.907+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c40b28a0 con 0x7fb0c413f710 2024-03-25T17:35:05.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1692487649 0 0) 0x7fb0b4003520 con 0x7fb0c40a98f0 2024-03-25T17:35:05.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 >> v1:172.21.15.134:6790/0 conn(0x7fb0c41430c0 legacy=0x7fb0c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:05.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 >> v1:172.21.15.134:6789/0 conn(0x7fb0c413f710 legacy=0x7fb0c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 1 monclient: found mon.noname-c 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c40b44c0 con 0x7fb0c40a98f0 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 10 monclient: _finish_auth 0 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb0b8001690 con 0x7fb0c40a98f0 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0c37fe640 10 monclient: dump: 2024-03-25T17:35:05.918 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0caa84640 5 monclient: authenticate success, global_id 4483 2024-03-25T17:35:05.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.916+0000 7fb0caa84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:05.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0b80017f0 con 0x7fb0c40a98f0 2024-03-25T17:35:05.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:05.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0c37fe640 1 -- 172.21.15.134:0/2283057958 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb0b8001c00 con 0x7fb0c40a98f0 2024-03-25T17:35:05.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:05.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:05.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0c37fe640 10 monclient: dump: 2024-03-25T17:35:05.934 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0caa84640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:05.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.933+0000 7fb0caa84640 4 set_mon_vals no callback set 2024-03-25T17:35:05.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.934+0000 7fb0caa84640 10 monclient: shutdown 2024-03-25T17:35:05.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.934+0000 7fb0caa84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.934+0000 7fb0caa84640 1 -- 172.21.15.134:0/2283057958 >> v1:172.21.15.159:6789/0 conn(0x7fb0c40a98f0 legacy=0x7fb0c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.934+0000 7fb0caa84640 1 -- 172.21.15.134:0/2283057958 shutdown_connections 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.934+0000 7fb0caa84640 1 -- 172.21.15.134:0/2283057958 >> 172.21.15.134:0/2283057958 conn(0x7fb0c40b7ea0 msgr2=0x7fb0c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.934+0000 7fb0caa84640 1 -- 172.21.15.134:0/2283057958 shutdown_connections 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.934+0000 7fb0caa84640 1 -- 172.21.15.134:0/2283057958 wait complete. 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 10 monclient: build_initial_monmap 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 10 monclient: monmap: 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:05.938 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 1 librados: starting msgr at 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 1 librados: starting objecter 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 1 Processor -- start 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 1 -- start start 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 1 librados: setting wanted keys 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 1 librados: calling monclient init 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.935+0000 7fb0caa84640 10 monclient: init 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 10 monclient(hunting): picked mon.noname-a con 0x7fb0c40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:05.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 10 monclient(hunting): picked mon.noname-c con 0x7fb0c413f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:05.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 10 monclient(hunting): picked mon.noname-b con 0x7fb0c41430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:05.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c4137ab0 con 0x7fb0c40a98f0 2024-03-25T17:35:05.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c4138bd0 con 0x7fb0c41430c0 2024-03-25T17:35:05.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c4138e40 con 0x7fb0c413f710 2024-03-25T17:35:05.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:05.940 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.936+0000 7fb0caa84640 10 monclient(hunting): authenticate will time out at 5659.199618s 2024-03-25T17:35:05.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.946+0000 7fb0c9021640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb0c41430c0 0x7fb0c41e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:58058/0 (socket says 172.21.15.134:58058) 2024-03-25T17:35:05.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.946+0000 7fb0c9021640 1 -- 172.21.15.134:0/3554702586 learned_addr learned my addr 172.21.15.134:0/3554702586 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:05.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.947+0000 7fb0c17fa640 10 client.?.objecter ms_handle_connect 0x7fb0c41430c0 2024-03-25T17:35:05.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.947+0000 7fb0c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:05.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.948+0000 7fb0c17fa640 10 client.?.objecter ms_handle_connect 0x7fb0c413f710 2024-03-25T17:35:05.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.948+0000 7fb0c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:05.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.958+0000 7fb0c17fa640 10 client.?.objecter ms_handle_connect 0x7fb0c40a98f0 2024-03-25T17:35:05.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:05.958+0000 7fb0c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:06.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.012+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595124541 0 0) 0x7fb0c4137ab0 con 0x7fb0c40a98f0 2024-03-25T17:35:06.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.012+0000 7fb0c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:06.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.012+0000 7fb0c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:06.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.012+0000 7fb0c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:06.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.012+0000 7fb0c17fa640 10 monclient(hunting): my global_id is 4464 2024-03-25T17:35:06.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.012+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0a8004eb0 con 0x7fb0c40a98f0 2024-03-25T17:35:06.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.033+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862235028 0 0) 0x7fb0c4138bd0 con 0x7fb0c41430c0 2024-03-25T17:35:06.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.033+0000 7fb0c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:06.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.033+0000 7fb0c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:06.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.033+0000 7fb0c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:06.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.033+0000 7fb0c17fa640 10 monclient(hunting): my global_id is 4486 2024-03-25T17:35:06.035 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.033+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c4137ab0 con 0x7fb0c41430c0 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.048+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4225761402 0 0) 0x7fb0a8004eb0 con 0x7fb0c40a98f0 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.048+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0c4138bd0 con 0x7fb0c40a98f0 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.048+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb0b8001bc0 con 0x7fb0c40a98f0 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.048+0000 7fb0c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.048+0000 7fb0c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.048+0000 7fb0c17fa640 10 monclient(hunting): dump: 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:06.049 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:06.050 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:06.050 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:06.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.048+0000 7fb0c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:06.059 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.059+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4242607213 0 0) 0x7fb0c4137ab0 con 0x7fb0c41430c0 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.059+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0a8004eb0 con 0x7fb0c41430c0 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.059+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb0bc001bd0 con 0x7fb0c41430c0 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.059+0000 7fb0c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.059+0000 7fb0c17fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.059+0000 7fb0c17fa640 10 monclient(hunting): dump: 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:06.060 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:06.061 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.076+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 662689013 0 0) 0x7fb0a8004eb0 con 0x7fb0c41430c0 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.076+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 >> v1:172.21.15.134:6790/0 conn(0x7fb0c413f710 legacy=0x7fb0c41df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.076+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 >> v1:172.21.15.134:6789/0 conn(0x7fb0c40a98f0 legacy=0x7fb0c4137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.076+0000 7fb0c17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.076+0000 7fb0c17fa640 1 monclient: found mon.b 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.076+0000 7fb0c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0c17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c4139120 con 0x7fb0c41430c0 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0c17fa640 10 monclient: _finish_auth 0 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0c17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:06.078 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 5 monclient: authenticate success, global_id 4486 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 10 monclient: _renew_subs 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c4139400 con 0x7fb0c41430c0 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 10 monclient: _renew_subs 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c40b7880 con 0x7fb0c41430c0 2024-03-25T17:35:06.079 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.077+0000 7fb0caa84640 1 librados: init done 2024-03-25T17:35:06.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.078+0000 7fb0caa84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:06.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.078+0000 7fb0caa84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:06.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.078+0000 7fb0caa84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:06.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.078+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c4137810 con 0x7fb0c41430c0 2024-03-25T17:35:06.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0bc0020a0 con 0x7fb0c41430c0 2024-03-25T17:35:06.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:06.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb0bc003b10 con 0x7fb0c41430c0 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c17fa640 10 monclient: dump: 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:06.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c2ffd640 4 set_mon_vals no callback set 2024-03-25T17:35:06.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.370+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fb0bc003d30 con 0x7fb0c41430c0 2024-03-25T17:35:06.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.371+0000 7fb0c8820640 10 monclient: get_auth_request con 0x7fb0a806ddf0 auth_method 0 2024-03-25T17:35:06.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.371+0000 7fb0c17fa640 10 client.4486.objecter ms_handle_connect 0x7fb0a806ddf0 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fb0bc054a70 con 0x7fb0c41430c0 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 10 client.4486.objecter ms_dispatch 0x7fb0c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0bc0503b0 con 0x7fb0c41430c0 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:06.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.378+0000 7fb0c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:06.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.632+0000 7fb0caa84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:06.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.632+0000 7fb0caa84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:06.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.632+0000 7fb0caa84640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:06.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.632+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0c4137c60 con 0x7fb0c41430c0 2024-03-25T17:35:06.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.654+0000 7fb0c17fa640 1 -- 172.21.15.134:0/3554702586 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fb0c4137c60 con 0x7fb0c41430c0 2024-03-25T17:35:06.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.654+0000 7fb0c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:06.655 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.654+0000 7fb0c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:06.655 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:35:06.655 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.656+0000 7fb0caa84640 10 librados: watch_flush enter 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.656+0000 7fb0caa84640 10 librados: watch_flush exit 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.656+0000 7fb0caa84640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.656+0000 7fb0caa84640 10 client.4486.objecter successfully canceled tick 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 >> v1:172.21.15.134:6800/610837766 conn(0x7fb0a806ddf0 legacy=0x7fb0a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 10 monclient: shutdown 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 >> v1:172.21.15.159:6789/0 conn(0x7fb0c41430c0 legacy=0x7fb0c41e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 shutdown_connections 2024-03-25T17:35:06.658 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 >> 172.21.15.134:0/3554702586 conn(0x7fb0c40b7ea0 msgr2=0x7fb0c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:06.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 shutdown_connections 2024-03-25T17:35:06.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 1 -- 172.21.15.134:0/3554702586 wait complete. 2024-03-25T17:35:06.659 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:06.657+0000 7fb0caa84640 1 librados: shutdown 2024-03-25T17:35:06.671 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:09.940 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:14.956 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:09.995 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:15.011 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:10.021 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:15.037 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:10.040 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:15.069 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:10.044 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:15.072 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:10.046 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:15.062 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:10.062 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:15.078 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:10.085 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:15.114 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:10.091 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:15.108 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:13.478 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:13.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.591+0000 7f3fed39c640 10 monclient: get_monmap_and_config 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.592+0000 7f3fed39c640 10 monclient: build_initial_monmap 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.592+0000 7f3fed39c640 10 monclient: monmap: 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:13.593014+0000 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:13.593014+0000 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:13.593 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.592+0000 7f3fed39c640 1 Processor -- start 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.592+0000 7f3fed39c640 1 -- start start 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.592+0000 7f3fed39c640 10 monclient: init 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe8142520 con 0x7f3fe813f710 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80b28a0 con 0x7f3fe81430c0 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80b2a30 con 0x7f3fe80a98f0 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:13.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.593+0000 7f3fed39c640 10 monclient(hunting): authenticate will time out at 5666.856350s 2024-03-25T17:35:13.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.610+0000 7f3fe6ffd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f3fe80a98f0 0x7f3fe80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:43912/0 (socket says 172.21.15.134:43912) 2024-03-25T17:35:13.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.610+0000 7f3fe6ffd640 1 -- 172.21.15.134:0/1301046504 learned_addr learned my addr 172.21.15.134:0/1301046504 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:13.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.626+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892017515 0 0) 0x7f3fe80b28a0 con 0x7f3fe81430c0 2024-03-25T17:35:13.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.626+0000 7f3fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:35:13.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.626+0000 7f3fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:13.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.626+0000 7f3fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:13.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.626+0000 7f3fe5ffb640 10 monclient(hunting): my global_id is 4489 2024-03-25T17:35:13.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.626+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd0003520 con 0x7f3fe81430c0 2024-03-25T17:35:13.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899785634 0 0) 0x7f3fe80b2a30 con 0x7f3fe80a98f0 2024-03-25T17:35:13.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:35:13.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:13.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:13.635 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): my global_id is 34454 2024-03-25T17:35:13.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe80b28a0 con 0x7f3fe80a98f0 2024-03-25T17:35:13.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094858128 0 0) 0x7f3fe8142520 con 0x7f3fe813f710 2024-03-25T17:35:13.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:35:13.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:13.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:13.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T17:35:13.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.635+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe80b2a30 con 0x7f3fe813f710 2024-03-25T17:35:13.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1296444269 0 0) 0x7f3fe80b28a0 con 0x7f3fe80a98f0 2024-03-25T17:35:13.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 >> v1:172.21.15.159:6789/0 conn(0x7f3fe81430c0 legacy=0x7f3fe814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:13.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 >> v1:172.21.15.134:6789/0 conn(0x7f3fe813f710 legacy=0x7f3fe81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:13.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:35:13.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 1 monclient: found mon.noname-b 2024-03-25T17:35:13.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:13.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe80b44c0 con 0x7f3fe80a98f0 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3fd8001670 con 0x7f3fe80a98f0 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fed39c640 5 monclient: authenticate success, global_id 34454 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fed39c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fe5ffb640 10 monclient: dump: 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.652 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:13.653 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:13.653 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:13.653 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:13.653 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:13.653 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:13.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.650+0000 7f3fed39c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fd80017d0 con 0x7f3fe80a98f0 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fe5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fe5ffb640 1 -- 172.21.15.134:0/1301046504 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3fd8001be0 con 0x7f3fe80a98f0 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fed39c640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fed39c640 4 set_mon_vals no callback set 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fe5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.674+0000 7f3fe5ffb640 10 monclient: dump: 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:13.675 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.676 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.676 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:13.676 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:13.676 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:13.676 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:13.676 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:13.676 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.675+0000 7f3fed39c640 10 monclient: shutdown 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.675+0000 7f3fed39c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.675+0000 7f3fed39c640 1 -- 172.21.15.134:0/1301046504 >> v1:172.21.15.134:6790/0 conn(0x7f3fe80a98f0 legacy=0x7f3fe80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.675+0000 7f3fed39c640 1 -- 172.21.15.134:0/1301046504 shutdown_connections 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.675+0000 7f3fed39c640 1 -- 172.21.15.134:0/1301046504 >> 172.21.15.134:0/1301046504 conn(0x7f3fe80b7ea0 msgr2=0x7f3fe80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.675+0000 7f3fed39c640 1 -- 172.21.15.134:0/1301046504 shutdown_connections 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.675+0000 7f3fed39c640 1 -- 172.21.15.134:0/1301046504 wait complete. 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 10 monclient: build_initial_monmap 2024-03-25T17:35:13.678 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 10 monclient: monmap: 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 1 librados: starting msgr at 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 1 librados: starting objecter 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 1 Processor -- start 2024-03-25T17:35:13.679 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 1 -- start start 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 1 librados: setting wanted keys 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 1 librados: calling monclient init 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.676+0000 7f3fed39c640 10 monclient: init 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 10 monclient(hunting): picked mon.noname-b con 0x7f3fe80a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 10 monclient(hunting): picked mon.noname-a con 0x7f3fe813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 10 monclient(hunting): picked mon.noname-c con 0x7f3fe81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80abc80 con 0x7f3fe813f710 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80abf60 con 0x7f3fe80a98f0 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe80ac240 con 0x7f3fe81430c0 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:13.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.677+0000 7f3fed39c640 10 monclient(hunting): authenticate will time out at 5666.940088s 2024-03-25T17:35:13.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.683+0000 7f3fe67fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f3fe813f710 0x7f3fe81365c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:58844/0 (socket says 172.21.15.134:58844) 2024-03-25T17:35:13.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.683+0000 7f3fe67fc640 1 -- 172.21.15.134:0/2865668633 learned_addr learned my addr 172.21.15.134:0/2865668633 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:13.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.690+0000 7f3fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f3fe813f710 2024-03-25T17:35:13.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.690+0000 7f3fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:13.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.692+0000 7f3fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f3fe81430c0 2024-03-25T17:35:13.692 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.692+0000 7f3fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:13.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.696+0000 7f3fcf7fe640 10 client.?.objecter ms_handle_connect 0x7f3fe80a98f0 2024-03-25T17:35:13.697 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.696+0000 7f3fcf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:13.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.699+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899584788 0 0) 0x7f3fe80abc80 con 0x7f3fe813f710 2024-03-25T17:35:13.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.699+0000 7f3fcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:13.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.699+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:13.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.699+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:13.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.699+0000 7f3fcf7fe640 10 monclient(hunting): my global_id is 4470 2024-03-25T17:35:13.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.699+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fd4004eb0 con 0x7f3fe813f710 2024-03-25T17:35:13.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.704+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3980115772 0 0) 0x7f3fe80abf60 con 0x7f3fe80a98f0 2024-03-25T17:35:13.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.704+0000 7f3fcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:13.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.704+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:13.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.704+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:13.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.704+0000 7f3fcf7fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T17:35:13.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.704+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe80abc80 con 0x7f3fe80a98f0 2024-03-25T17:35:13.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.707+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100211535 0 0) 0x7f3fe80ac240 con 0x7f3fe81430c0 2024-03-25T17:35:13.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.707+0000 7f3fcf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:13.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.707+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:13.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.707+0000 7f3fcf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:13.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.707+0000 7f3fcf7fe640 10 monclient(hunting): my global_id is 34457 2024-03-25T17:35:13.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.707+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe80abf60 con 0x7f3fe81430c0 2024-03-25T17:35:13.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.721+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2776710736 0 0) 0x7f3fd4004eb0 con 0x7f3fe813f710 2024-03-25T17:35:13.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.721+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe80ac240 con 0x7f3fe813f710 2024-03-25T17:35:13.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.721+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3fdc001a10 con 0x7f3fe813f710 2024-03-25T17:35:13.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.721+0000 7f3fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:13.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.721+0000 7f3fcf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:13.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.721+0000 7f3fcf7fe640 10 monclient(hunting): dump: 2024-03-25T17:35:13.722 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:13.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.721+0000 7f3fcf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:13.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.735+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3821491306 0 0) 0x7f3fe80abf60 con 0x7f3fe81430c0 2024-03-25T17:35:13.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.735+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fd4004eb0 con 0x7f3fe81430c0 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.735+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3fe0001a30 con 0x7f3fe81430c0 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.735+0000 7f3fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.735+0000 7f3fcf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.735+0000 7f3fcf7fe640 10 monclient(hunting): dump: 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:13.736 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.737+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1917348571 0 0) 0x7f3fe80abc80 con 0x7f3fe80a98f0 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.737+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3fe80abf60 con 0x7f3fe80a98f0 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.737+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3fd8001ae0 con 0x7f3fe80a98f0 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.737+0000 7f3fcf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.737+0000 7f3fcf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.737+0000 7f3fcf7fe640 10 monclient(hunting): dump: 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:13.738 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:13.739 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:13.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4060921585 0 0) 0x7f3fe80ac240 con 0x7f3fe813f710 2024-03-25T17:35:13.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 >> v1:172.21.15.134:6790/0 conn(0x7f3fe81430c0 legacy=0x7f3fe81e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 >> v1:172.21.15.159:6789/0 conn(0x7f3fe80a98f0 legacy=0x7f3fe80ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 1 monclient: found mon.a 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe8138520 con 0x7f3fe813f710 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 10 monclient: _finish_auth 0 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fcf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 5 monclient: authenticate success, global_id 4470 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 10 monclient: _renew_subs 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:13.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe8137f50 con 0x7f3fe813f710 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 10 monclient: _renew_subs 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe8138e20 con 0x7f3fe813f710 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.745+0000 7f3fed39c640 1 librados: init done 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.746+0000 7f3fed39c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.746+0000 7f3fed39c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.746+0000 7f3fed39c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:13.747 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:13.746+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe80b3ef0 con 0x7f3fe813f710 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3fdc001c70 con 0x7f3fe813f710 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fcf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f3fdc003a60 con 0x7f3fe813f710 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fcf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fcf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fcf7fe640 10 monclient: dump: 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:14.087 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fe57fa640 4 set_mon_vals no callback set 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.086+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f3fdc003c80 con 0x7f3fe813f710 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.087+0000 7f3fe6ffd640 10 monclient: get_auth_request con 0x7f3fd406e0f0 auth_method 0 2024-03-25T17:35:14.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.087+0000 7f3fcf7fe640 10 client.4470.objecter ms_handle_connect 0x7f3fd406e0f0 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f3fdc054860 con 0x7f3fe813f710 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 10 client.4470.objecter ms_dispatch 0x7f3fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3fdc050370 con 0x7f3fe813f710 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:14.093 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.092+0000 7f3fcf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:14.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.336+0000 7f3fed39c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:14.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.336+0000 7f3fed39c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:14.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.336+0000 7f3fed39c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:14.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.336+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fe8133ec0 con 0x7f3fe813f710 2024-03-25T17:35:14.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.377+0000 7f3fcf7fe640 1 -- 172.21.15.134:0/2865668633 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f3fe8133ec0 con 0x7f3fe813f710 2024-03-25T17:35:14.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.377+0000 7f3fcf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:14.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.377+0000 7f3fcf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:14.378 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:35:14.378 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 10 librados: watch_flush enter 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 10 librados: watch_flush exit 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 10 client.4470.objecter successfully canceled tick 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 >> v1:172.21.15.134:6800/610837766 conn(0x7f3fd406e0f0 legacy=0x7f3fd40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 10 monclient: shutdown 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 >> v1:172.21.15.134:6789/0 conn(0x7f3fe813f710 legacy=0x7f3fe81365c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 shutdown_connections 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.379+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 >> 172.21.15.134:0/2865668633 conn(0x7f3fe80b7ea0 msgr2=0x7f3fe8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.380+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 shutdown_connections 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.380+0000 7f3fed39c640 1 -- 172.21.15.134:0/2865668633 wait complete. 2024-03-25T17:35:14.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:14.380+0000 7f3fed39c640 1 librados: shutdown 2024-03-25T17:35:14.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:21.198 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:21.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.309+0000 7f2a12686640 10 monclient: get_monmap_and_config 2024-03-25T17:35:21.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.309+0000 7f2a12686640 10 monclient: build_initial_monmap 2024-03-25T17:35:21.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.309+0000 7f2a12686640 10 monclient: monmap: 2024-03-25T17:35:21.310 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:21.310 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:21.310828+0000 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:21.310828+0000 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:21.311 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:21.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.310+0000 7f2a12686640 1 Processor -- start 2024-03-25T17:35:21.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.310+0000 7f2a12686640 1 -- start start 2024-03-25T17:35:21.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.310+0000 7f2a12686640 10 monclient: init 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c0b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0af940 con 0x7f2a0c14be90 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0b0a70 con 0x7f2a0c0b1730 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0b0d50 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:21.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.311+0000 7f2a12686640 10 monclient(hunting): authenticate will time out at 5674.574147s 2024-03-25T17:35:21.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.315+0000 7f2a10c23640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2a0c14be90 0x7f2a0c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:54942/0 (socket says 172.21.15.134:54942) 2024-03-25T17:35:21.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.315+0000 7f2a10c23640 1 -- 172.21.15.134:0/164824710 learned_addr learned my addr 172.21.15.134:0/164824710 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:21.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.324+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789204949 0 0) 0x7f2a0c0af940 con 0x7f2a0c14be90 2024-03-25T17:35:21.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.324+0000 7f2a0affd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:21.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.324+0000 7f2a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:21.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.324+0000 7f2a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:21.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.324+0000 7f2a0affd640 10 monclient(hunting): my global_id is 4473 2024-03-25T17:35:21.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.324+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f4003520 con 0x7f2a0c14be90 2024-03-25T17:35:21.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.333+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001416966 0 0) 0x7f2a0c0b0d50 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.333+0000 7f2a0affd640 10 monclient(hunting): _negotiate 2024-03-25T17:35:21.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.333+0000 7f2a0affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:21.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.333+0000 7f2a0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:21.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.333+0000 7f2a0affd640 10 monclient(hunting): my global_id is 34460 2024-03-25T17:35:21.333 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.333+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c0af940 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2902971675 0 0) 0x7f29f4003520 con 0x7f2a0c14be90 2024-03-25T17:35:21.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 >> v1:172.21.15.134:6790/0 conn(0x7f2a0c0b4d60 legacy=0x7f2a0c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 >> v1:172.21.15.159:6789/0 conn(0x7f2a0c0b1730 legacy=0x7f2a0c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 10 monclient: _finish_hunting 0 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 1 monclient: found mon.noname-a 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0b1030 con 0x7f2a0c14be90 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 10 monclient: _finish_auth 0 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 10 monclient: _check_auth_tickets 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2a04001a10 con 0x7f2a0c14be90 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a0affd640 10 monclient: dump: 2024-03-25T17:35:21.337 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a12686640 5 monclient: authenticate success, global_id 4473 2024-03-25T17:35:21.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.336+0000 7f2a12686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:21.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a04001b70 con 0x7f2a0c14be90 2024-03-25T17:35:21.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:21.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a0affd640 1 -- 172.21.15.134:0/164824710 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2a04001f80 con 0x7f2a0c14be90 2024-03-25T17:35:21.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a12686640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:21.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a12686640 4 set_mon_vals no callback set 2024-03-25T17:35:21.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:21.339 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.338+0000 7f2a0affd640 10 monclient: dump: 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:21.340 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.339+0000 7f2a12686640 10 monclient: shutdown 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.339+0000 7f2a12686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.339+0000 7f2a12686640 1 -- 172.21.15.134:0/164824710 >> v1:172.21.15.134:6789/0 conn(0x7f2a0c14be90 legacy=0x7f2a0c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.339+0000 7f2a12686640 1 -- 172.21.15.134:0/164824710 shutdown_connections 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.339+0000 7f2a12686640 1 -- 172.21.15.134:0/164824710 >> 172.21.15.134:0/164824710 conn(0x7f2a0c0617a0 msgr2=0x7f2a0c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.339+0000 7f2a12686640 1 -- 172.21.15.134:0/164824710 shutdown_connections 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.339+0000 7f2a12686640 1 -- 172.21.15.134:0/164824710 wait complete. 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.340+0000 7f2a12686640 10 monclient: build_initial_monmap 2024-03-25T17:35:21.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.340+0000 7f2a12686640 10 monclient: monmap: 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.340+0000 7f2a12686640 1 librados: starting msgr at 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.340+0000 7f2a12686640 1 librados: starting objecter 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.340+0000 7f2a12686640 1 Processor -- start 2024-03-25T17:35:21.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.341+0000 7f2a12686640 1 -- start start 2024-03-25T17:35:21.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.341+0000 7f2a12686640 1 librados: setting wanted keys 2024-03-25T17:35:21.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.341+0000 7f2a12686640 1 librados: calling monclient init 2024-03-25T17:35:21.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.341+0000 7f2a12686640 10 monclient: init 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 10 monclient(hunting): picked mon.noname-c con 0x7f2a0c0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 10 monclient(hunting): picked mon.noname-b con 0x7f2a0c0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 10 monclient(hunting): picked mon.noname-a con 0x7f2a0c14be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c1d87c0 con 0x7f2a0c14be90 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c138820 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c1389b0 con 0x7f2a0c0b1730 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:21.345 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.342+0000 7f2a12686640 10 monclient(hunting): authenticate will time out at 5674.604896s 2024-03-25T17:35:21.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.354+0000 7f2a10c23640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2a0c14be90 0x7f2a0c1e22a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:54958/0 (socket says 172.21.15.134:54958) 2024-03-25T17:35:21.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.354+0000 7f2a10c23640 1 -- 172.21.15.134:0/3278731668 learned_addr learned my addr 172.21.15.134:0/3278731668 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:21.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.358+0000 7f2a08ff9640 10 client.?.objecter ms_handle_connect 0x7f2a0c14be90 2024-03-25T17:35:21.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.358+0000 7f2a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:21.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.361+0000 7f2a08ff9640 10 client.?.objecter ms_handle_connect 0x7f2a0c0b1730 2024-03-25T17:35:21.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.361+0000 7f2a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:21.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.363+0000 7f2a08ff9640 10 client.?.objecter ms_handle_connect 0x7f2a0c0b4d60 2024-03-25T17:35:21.364 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.363+0000 7f2a08ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:21.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.366+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542130632 0 0) 0x7f2a0c1d87c0 con 0x7f2a0c14be90 2024-03-25T17:35:21.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.366+0000 7f2a08ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:21.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.366+0000 7f2a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:21.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.366+0000 7f2a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:21.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.366+0000 7f2a08ff9640 10 monclient(hunting): my global_id is 4476 2024-03-25T17:35:21.367 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.366+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8004eb0 con 0x7f2a0c14be90 2024-03-25T17:35:21.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.367+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001033440 0 0) 0x7f2a0c138820 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.367+0000 7f2a08ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:21.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.367+0000 7f2a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:21.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.367+0000 7f2a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:21.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.367+0000 7f2a08ff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T17:35:21.368 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.367+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c1d87c0 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.375+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648694009 0 0) 0x7f2a0c1389b0 con 0x7f2a0c0b1730 2024-03-25T17:35:21.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.375+0000 7f2a08ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:35:21.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.375+0000 7f2a08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:21.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.375+0000 7f2a08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:21.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.375+0000 7f2a08ff9640 10 monclient(hunting): my global_id is 34463 2024-03-25T17:35:21.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.375+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a0c138820 con 0x7f2a0c0b1730 2024-03-25T17:35:21.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.377+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4212895734 0 0) 0x7f29f8004eb0 con 0x7f2a0c14be90 2024-03-25T17:35:21.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.377+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a0c1389b0 con 0x7f2a0c14be90 2024-03-25T17:35:21.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.377+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2a04001ec0 con 0x7f2a0c14be90 2024-03-25T17:35:21.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.377+0000 7f2a08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:21.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.377+0000 7f2a08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.377+0000 7f2a08ff9640 10 monclient(hunting): dump: 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:21.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.377+0000 7f2a08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:21.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.381+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 292616316 0 0) 0x7f2a0c1d87c0 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.381+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29f8004eb0 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.381+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2a000018b0 con 0x7f2a0c0b4d60 2024-03-25T17:35:21.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.381+0000 7f2a08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:21.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.381+0000 7f2a08ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.381+0000 7f2a08ff9640 10 monclient(hunting): dump: 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:21.383 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:21.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3689460918 0 0) 0x7f2a0c1389b0 con 0x7f2a0c14be90 2024-03-25T17:35:21.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 >> v1:172.21.15.134:6790/0 conn(0x7f2a0c0b1730 legacy=0x7f2a0c138060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:21.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 >> v1:172.21.15.159:6789/0 conn(0x7f2a0c0b4d60 legacy=0x7f2a0c1d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:21.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 1 monclient: found mon.a 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c1405f0 con 0x7f2a0c14be90 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 10 monclient: _finish_auth 0 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 5 monclient: authenticate success, global_id 4476 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 10 monclient: _renew_subs 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c1407e0 con 0x7f2a0c14be90 2024-03-25T17:35:21.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 10 client.4476.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:21.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 10 monclient: _renew_subs 2024-03-25T17:35:21.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:21.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1e3ba0 con 0x7f2a0c14be90 2024-03-25T17:35:21.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.384+0000 7f2a12686640 1 librados: init done 2024-03-25T17:35:21.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.385+0000 7f2a12686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:21.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.385+0000 7f2a12686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:21.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.385+0000 7f2a12686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:21.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.385+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c1d7f00 con 0x7f2a0c14be90 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.682+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2a04004e10 con 0x7f2a0c14be90 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.682+0000 7f2a08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.682+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2a04002100 con 0x7f2a0c14be90 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.682+0000 7f2a08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.682+0000 7f2a08ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.682+0000 7f2a08ff9640 10 monclient: dump: 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:21.683 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:21.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.682+0000 7f2a0a7fc640 4 set_mon_vals no callback set 2024-03-25T17:35:21.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.683+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f2a0401af90 con 0x7f2a0c14be90 2024-03-25T17:35:21.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.683+0000 7f2a0bfff640 10 monclient: get_auth_request con 0x7f29f806df70 auth_method 0 2024-03-25T17:35:21.685 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.684+0000 7f2a08ff9640 10 client.4476.objecter ms_handle_connect 0x7f29f806df70 2024-03-25T17:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f2a04054b70 con 0x7f2a0c14be90 2024-03-25T17:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 10 client.4476.objecter ms_dispatch 0x7f2a0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 3 client.4476.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 3 client.4476.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 20 client.4476.objecter dump_active .. 0 homeless 2024-03-25T17:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2a040504b0 con 0x7f2a0c14be90 2024-03-25T17:35:21.690 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:21.691 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.689+0000 7f2a08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:21.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.941+0000 7f2a12686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:21.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.941+0000 7f2a12686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:21.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.941+0000 7f2a12686640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:35:21.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.941+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a0c0b5ac0 con 0x7f2a0c14be90 2024-03-25T17:35:21.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.972+0000 7f2a08ff9640 1 -- 172.21.15.134:0/3278731668 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f2a040525b0 con 0x7f2a0c14be90 2024-03-25T17:35:21.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.972+0000 7f2a08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:21.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.972+0000 7f2a08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:21.973 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:35:21.973 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:21.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 10 librados: watch_flush enter 2024-03-25T17:35:21.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 10 librados: watch_flush exit 2024-03-25T17:35:21.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 20 client.4476.objecter shutdown clearing up homeless session... 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 10 client.4476.objecter successfully canceled tick 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 >> v1:172.21.15.134:6800/610837766 conn(0x7f29f806df70 legacy=0x7f29f8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 10 monclient: shutdown 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 >> v1:172.21.15.134:6789/0 conn(0x7f2a0c14be90 legacy=0x7f2a0c1e22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.974+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 shutdown_connections 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.975+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 >> 172.21.15.134:0/3278731668 conn(0x7f2a0c0617a0 msgr2=0x7f2a0c0b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.975+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 shutdown_connections 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.975+0000 7f2a12686640 1 -- 172.21.15.134:0/3278731668 wait complete. 2024-03-25T17:35:21.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:21.975+0000 7f2a12686640 1 librados: shutdown 2024-03-25T17:35:21.988 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:28.796 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:28.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.914+0000 7f77c5c06640 10 monclient: get_monmap_and_config 2024-03-25T17:35:28.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.914+0000 7f77c5c06640 10 monclient: build_initial_monmap 2024-03-25T17:35:28.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.914+0000 7f77c5c06640 10 monclient: monmap: 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:28.915857+0000 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:28.915857+0000 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:28.916 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:28.917 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:28.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.915+0000 7f77c5c06640 1 Processor -- start 2024-03-25T17:35:28.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.915+0000 7f77c5c06640 1 -- start start 2024-03-25T17:35:28.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.915+0000 7f77c5c06640 10 monclient: init 2024-03-25T17:35:28.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 10 monclient(hunting): picked mon.noname-c con 0x7f77c00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 10 monclient(hunting): picked mon.noname-b con 0x7f77c013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 10 monclient(hunting): picked mon.noname-a con 0x7f77c01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c0142520 con 0x7f77c01430c0 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c00b28a0 con 0x7f77c00a98f0 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c00b2a30 con 0x7f77c013f710 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:28.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.916+0000 7f77c5c06640 10 monclient(hunting): authenticate will time out at 5682.179294s 2024-03-25T17:35:28.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.919+0000 7f77bf7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f77c00a98f0 0x7f77c00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54844/0 (socket says 172.21.15.134:54844) 2024-03-25T17:35:28.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.919+0000 7f77bf7fe640 1 -- 172.21.15.134:0/2087019457 learned_addr learned my addr 172.21.15.134:0/2087019457 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:28.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.926+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371758509 0 0) 0x7f77c00b28a0 con 0x7f77c00a98f0 2024-03-25T17:35:28.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.926+0000 7f77be7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:35:28.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.926+0000 7f77be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:28.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.926+0000 7f77be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:28.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.926+0000 7f77be7fc640 10 monclient(hunting): my global_id is 4501 2024-03-25T17:35:28.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.926+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a8003520 con 0x7f77c00a98f0 2024-03-25T17:35:28.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 78832260 0 0) 0x7f77a8003520 con 0x7f77c00a98f0 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 >> v1:172.21.15.134:6790/0 conn(0x7f77c013f710 legacy=0x7f77c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 >> v1:172.21.15.134:6789/0 conn(0x7f77c01430c0 legacy=0x7f77c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 1 monclient: found mon.noname-c 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c00b44c0 con 0x7f77c00a98f0 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 10 monclient: _finish_auth 0 2024-03-25T17:35:28.945 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f77b0001690 con 0x7f77c00a98f0 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77c5c06640 5 monclient: authenticate success, global_id 4501 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77c5c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77be7fc640 10 monclient: dump: 2024-03-25T17:35:28.946 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:28.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.943+0000 7f77c5c06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77b00017f0 con 0x7f77c00a98f0 2024-03-25T17:35:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77be7fc640 1 -- 172.21.15.134:0/2087019457 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f77b0001c00 con 0x7f77c00a98f0 2024-03-25T17:35:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77c5c06640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77c5c06640 4 set_mon_vals no callback set 2024-03-25T17:35:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:28.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.949+0000 7f77be7fc640 10 monclient: dump: 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:28.951 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:28.952 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:28.952 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:28.954 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 10 monclient: shutdown 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 1 -- 172.21.15.134:0/2087019457 >> v1:172.21.15.159:6789/0 conn(0x7f77c00a98f0 legacy=0x7f77c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 1 -- 172.21.15.134:0/2087019457 shutdown_connections 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 1 -- 172.21.15.134:0/2087019457 >> 172.21.15.134:0/2087019457 conn(0x7f77c00b7ea0 msgr2=0x7f77c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 1 -- 172.21.15.134:0/2087019457 shutdown_connections 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 1 -- 172.21.15.134:0/2087019457 wait complete. 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 10 monclient: build_initial_monmap 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.950+0000 7f77c5c06640 10 monclient: monmap: 2024-03-25T17:35:28.955 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:28.956 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.951+0000 7f77c5c06640 1 librados: starting msgr at 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.951+0000 7f77c5c06640 1 librados: starting objecter 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.951+0000 7f77c5c06640 1 Processor -- start 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.951+0000 7f77c5c06640 1 -- start start 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.951+0000 7f77c5c06640 1 librados: setting wanted keys 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.951+0000 7f77c5c06640 1 librados: calling monclient init 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.951+0000 7f77c5c06640 10 monclient: init 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 10 monclient(hunting): picked mon.noname-a con 0x7f77c00a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:28.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 10 monclient(hunting): picked mon.noname-c con 0x7f77c013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:28.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 10 monclient(hunting): picked mon.noname-b con 0x7f77c01430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:28.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c006f420 con 0x7f77c00a98f0 2024-03-25T17:35:28.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c0070270 con 0x7f77c01430c0 2024-03-25T17:35:28.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c0070550 con 0x7f77c013f710 2024-03-25T17:35:28.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:28.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.952+0000 7f77c5c06640 10 monclient(hunting): authenticate will time out at 5682.215335s 2024-03-25T17:35:28.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.957+0000 7f77bffff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f77c01430c0 0x7f77c01dc1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54892/0 (socket says 172.21.15.134:54892) 2024-03-25T17:35:28.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.957+0000 7f77bffff640 1 -- 172.21.15.134:0/1159269521 learned_addr learned my addr 172.21.15.134:0/1159269521 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:28.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.958+0000 7f77a7fff640 10 client.?.objecter ms_handle_connect 0x7f77c01430c0 2024-03-25T17:35:28.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.958+0000 7f77a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:28.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.967+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017647334 0 0) 0x7f77c0070270 con 0x7f77c01430c0 2024-03-25T17:35:28.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.967+0000 7f77a7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:28.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.967+0000 7f77a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:28.968 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.967+0000 7f77a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:28.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.967+0000 7f77a7fff640 10 monclient(hunting): my global_id is 4504 2024-03-25T17:35:28.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.967+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77ac004eb0 con 0x7f77c01430c0 2024-03-25T17:35:28.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.978+0000 7f77a7fff640 10 client.?.objecter ms_handle_connect 0x7f77c013f710 2024-03-25T17:35:28.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.978+0000 7f77a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:28.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.980+0000 7f77a7fff640 10 client.?.objecter ms_handle_connect 0x7f77c00a98f0 2024-03-25T17:35:28.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.980+0000 7f77a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:28.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.994+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169236417 0 0) 0x7f77c0070550 con 0x7f77c013f710 2024-03-25T17:35:28.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.994+0000 7f77a7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:28.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.994+0000 7f77a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:28.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.994+0000 7f77a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:28.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.994+0000 7f77a7fff640 10 monclient(hunting): my global_id is 34469 2024-03-25T17:35:28.995 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.994+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c0070270 con 0x7f77c013f710 2024-03-25T17:35:29.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.999+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190567274 0 0) 0x7f77c006f420 con 0x7f77c00a98f0 2024-03-25T17:35:29.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.999+0000 7f77a7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:29.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.999+0000 7f77a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:29.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.999+0000 7f77a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:29.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.999+0000 7f77a7fff640 10 monclient(hunting): my global_id is 4482 2024-03-25T17:35:29.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:28.999+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c0070550 con 0x7f77c00a98f0 2024-03-25T17:35:29.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.004+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2397321839 0 0) 0x7f77ac004eb0 con 0x7f77c01430c0 2024-03-25T17:35:29.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.004+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77c006f420 con 0x7f77c01430c0 2024-03-25T17:35:29.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.004+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f77b8001c90 con 0x7f77c01430c0 2024-03-25T17:35:29.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.004+0000 7f77a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:29.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.004+0000 7f77a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.004+0000 7f77a7fff640 10 monclient(hunting): dump: 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:29.006 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:29.007 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:29.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.004+0000 7f77a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:29.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.011+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2194340206 0 0) 0x7f77c0070270 con 0x7f77c013f710 2024-03-25T17:35:29.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.011+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77ac004eb0 con 0x7f77c013f710 2024-03-25T17:35:29.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.011+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f77b40018b0 con 0x7f77c013f710 2024-03-25T17:35:29.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.011+0000 7f77a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:29.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.011+0000 7f77a7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.011+0000 7f77a7fff640 10 monclient(hunting): dump: 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:29.013 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:29.014 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2199302087 0 0) 0x7f77ac004eb0 con 0x7f77c013f710 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 >> v1:172.21.15.159:6789/0 conn(0x7f77c01430c0 legacy=0x7f77c01dc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 >> v1:172.21.15.134:6789/0 conn(0x7f77c00a98f0 legacy=0x7f77c006e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 1 monclient: found mon.c 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:29.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c006dea0 con 0x7f77c013f710 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 10 monclient: _finish_auth 0 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 5 monclient: authenticate success, global_id 34469 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 10 monclient: _renew_subs 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c006f020 con 0x7f77c013f710 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 10 client.34469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:29.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 10 monclient: _renew_subs 2024-03-25T17:35:29.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:29.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c01e7550 con 0x7f77c013f710 2024-03-25T17:35:29.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.019+0000 7f77c5c06640 1 librados: init done 2024-03-25T17:35:29.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.021+0000 7f77c5c06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:29.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.021+0000 7f77c5c06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:29.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.021+0000 7f77c5c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:29.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.021+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c01db7d0 con 0x7f77c013f710 2024-03-25T17:35:29.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.327+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77b4003610 con 0x7f77c013f710 2024-03-25T17:35:29.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.327+0000 7f77a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:29.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.327+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f77b4003780 con 0x7f77c013f710 2024-03-25T17:35:29.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.327+0000 7f77a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:29.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.327+0000 7f77a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:29.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.327+0000 7f77a7fff640 10 monclient: dump: 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:29.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.327+0000 7f77bdffb640 4 set_mon_vals no callback set 2024-03-25T17:35:29.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.328+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f77b4019750 con 0x7f77c013f710 2024-03-25T17:35:29.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.329+0000 7f77bf7fe640 10 monclient: get_auth_request con 0x7f77ac06df70 auth_method 0 2024-03-25T17:35:29.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.329+0000 7f77a7fff640 10 client.34469.objecter ms_handle_connect 0x7f77ac06df70 2024-03-25T17:35:29.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.334+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f77b4053420 con 0x7f77c013f710 2024-03-25T17:35:29.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.334+0000 7f77a7fff640 10 client.34469.objecter ms_dispatch 0x7f77c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:29.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.334+0000 7f77a7fff640 3 client.34469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:29.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.334+0000 7f77a7fff640 3 client.34469.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:29.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.334+0000 7f77a7fff640 20 client.34469.objecter dump_active .. 0 homeless 2024-03-25T17:35:29.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.334+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77b404ef30 con 0x7f77c013f710 2024-03-25T17:35:29.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.335+0000 7f77a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:29.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.335+0000 7f77a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:29.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.555+0000 7f77c5c06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:29.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.555+0000 7f77c5c06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:29.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.555+0000 7f77c5c06640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:29.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.555+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77c0074cc0 con 0x7f77c013f710 2024-03-25T17:35:29.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.603+0000 7f77a7fff640 1 -- 172.21.15.134:0/1159269521 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f77b4050e60 con 0x7f77c013f710 2024-03-25T17:35:29.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.603+0000 7f77a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:29.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.603+0000 7f77a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:29.605 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:35:29.605 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.605+0000 7f77c5c06640 10 librados: watch_flush enter 2024-03-25T17:35:29.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.605+0000 7f77c5c06640 10 librados: watch_flush exit 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.605+0000 7f77c5c06640 20 client.34469.objecter shutdown clearing up homeless session... 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.605+0000 7f77c5c06640 10 client.34469.objecter successfully canceled tick 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 >> v1:172.21.15.134:6800/610837766 conn(0x7f77ac06df70 legacy=0x7f77ac070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 10 monclient: shutdown 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 >> v1:172.21.15.134:6790/0 conn(0x7f77c013f710 legacy=0x7f77c01dbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 shutdown_connections 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 >> 172.21.15.134:0/1159269521 conn(0x7f77c00b7ea0 msgr2=0x7f77c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 shutdown_connections 2024-03-25T17:35:29.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 1 -- 172.21.15.134:0/1159269521 wait complete. 2024-03-25T17:35:29.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:29.606+0000 7f77c5c06640 1 librados: shutdown 2024-03-25T17:35:29.618 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:30.050 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:35.066 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.104 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:35.120 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.141 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:35.157 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.170 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:35.199 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.175 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:35.191 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.178 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:35.207 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.182 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:35.199 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.205 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:35.234 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:30.209 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:35.225 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:36.424 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:36.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.541+0000 7fe38a172640 10 monclient: get_monmap_and_config 2024-03-25T17:35:36.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.541+0000 7fe38a172640 10 monclient: build_initial_monmap 2024-03-25T17:35:36.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.541+0000 7fe38a172640 10 monclient: monmap: 2024-03-25T17:35:36.542 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:36.542 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:36.542 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:36.542447+0000 2024-03-25T17:35:36.542 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:36.542447+0000 2024-03-25T17:35:36.543 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:36.543 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:36.543 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:36.543 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:36.543 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:36.543 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:36.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.542+0000 7fe38a172640 1 Processor -- start 2024-03-25T17:35:36.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.542+0000 7fe38a172640 1 -- start start 2024-03-25T17:35:36.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.542+0000 7fe38a172640 10 monclient: init 2024-03-25T17:35:36.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.542+0000 7fe38a172640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:36.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:35:36.544 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 10 monclient(hunting): picked mon.noname-a con 0x7fe3840a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:36.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 10 monclient(hunting): picked mon.noname-c con 0x7fe38413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:36.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 10 monclient(hunting): picked mon.noname-b con 0x7fe3841430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:36.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe384142520 con 0x7fe3840a98f0 2024-03-25T17:35:36.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3840b28a0 con 0x7fe38413f710 2024-03-25T17:35:36.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3840b2a30 con 0x7fe3841430c0 2024-03-25T17:35:36.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:36.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.543+0000 7fe38a172640 10 monclient(hunting): authenticate will time out at 5689.805883s 2024-03-25T17:35:36.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.549+0000 7fe382ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe38413f710 0x7fe3841429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54244/0 (socket says 172.21.15.134:54244) 2024-03-25T17:35:36.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.549+0000 7fe382ffd640 1 -- 172.21.15.134:0/3793584347 learned_addr learned my addr 172.21.15.134:0/3793584347 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:36.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.563+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550785998 0 0) 0x7fe3840b28a0 con 0x7fe38413f710 2024-03-25T17:35:36.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.563+0000 7fe3827fc640 10 monclient(hunting): _negotiate 2024-03-25T17:35:36.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.563+0000 7fe3827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:36.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.563+0000 7fe3827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:36.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.563+0000 7fe3827fc640 10 monclient(hunting): my global_id is 4507 2024-03-25T17:35:36.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.563+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36c003520 con 0x7fe38413f710 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874203888 0 0) 0x7fe36c003520 con 0x7fe38413f710 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 >> v1:172.21.15.134:6790/0 conn(0x7fe3841430c0 legacy=0x7fe38414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 >> v1:172.21.15.134:6789/0 conn(0x7fe3840a98f0 legacy=0x7fe3840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 10 monclient: _finish_hunting 0 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 1 monclient: found mon.noname-c 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3840b44c0 con 0x7fe38413f710 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 10 monclient: _finish_auth 0 2024-03-25T17:35:36.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 10 monclient: _check_auth_tickets 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe378001690 con 0x7fe38413f710 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe38a172640 5 monclient: authenticate success, global_id 4507 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe38a172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe3827fc640 10 monclient: dump: 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:36.582 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:36.583 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:36.583 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:36.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.580+0000 7fe38a172640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3780017f0 con 0x7fe38413f710 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe3827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe3827fc640 1 -- 172.21.15.134:0/3793584347 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe378001c40 con 0x7fe38413f710 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe38a172640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe38a172640 4 set_mon_vals no callback set 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe3827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe3827fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.591+0000 7fe3827fc640 10 monclient: dump: 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.592 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:36.593 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:36.593 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:36.593 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:36.593 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:36.593 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:36.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.592+0000 7fe38a172640 10 monclient: shutdown 2024-03-25T17:35:36.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.592+0000 7fe38a172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:36.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.592+0000 7fe38a172640 1 -- 172.21.15.134:0/3793584347 >> v1:172.21.15.159:6789/0 conn(0x7fe38413f710 legacy=0x7fe3841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:36.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.592+0000 7fe38a172640 1 -- 172.21.15.134:0/3793584347 shutdown_connections 2024-03-25T17:35:36.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.592+0000 7fe38a172640 1 -- 172.21.15.134:0/3793584347 >> 172.21.15.134:0/3793584347 conn(0x7fe3840b7ea0 msgr2=0x7fe3840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:36.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.592+0000 7fe38a172640 1 -- 172.21.15.134:0/3793584347 shutdown_connections 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.592+0000 7fe38a172640 1 -- 172.21.15.134:0/3793584347 wait complete. 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 10 monclient: build_initial_monmap 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 10 monclient: monmap: 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:36.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 1 librados: starting msgr at 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 1 librados: starting objecter 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 1 Processor -- start 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 1 -- start start 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 1 librados: setting wanted keys 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 1 librados: calling monclient init 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.593+0000 7fe38a172640 10 monclient: init 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 10 monclient(hunting): picked mon.noname-c con 0x7fe3840a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 10 monclient(hunting): picked mon.noname-a con 0x7fe38413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 10 monclient(hunting): picked mon.noname-b con 0x7fe3841430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3840b1870 con 0x7fe38413f710 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3840ac910 con 0x7fe3841430c0 2024-03-25T17:35:36.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3840abb00 con 0x7fe3840a98f0 2024-03-25T17:35:36.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:36.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.594+0000 7fe38a172640 10 monclient(hunting): authenticate will time out at 5689.857421s 2024-03-25T17:35:36.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.600+0000 7fe3837fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe3840a98f0 0x7fe384138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:40356/0 (socket says 172.21.15.134:40356) 2024-03-25T17:35:36.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.600+0000 7fe3837fe640 1 -- 172.21.15.134:0/3793633897 learned_addr learned my addr 172.21.15.134:0/3793633897 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:36.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.601+0000 7fe36bfff640 10 client.?.objecter ms_handle_connect 0x7fe3840a98f0 2024-03-25T17:35:36.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.601+0000 7fe36bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:36.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.610+0000 7fe36bfff640 10 client.?.objecter ms_handle_connect 0x7fe3841430c0 2024-03-25T17:35:36.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.610+0000 7fe36bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:36.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.617+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601965548 0 0) 0x7fe3840abb00 con 0x7fe3840a98f0 2024-03-25T17:35:36.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.617+0000 7fe36bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:36.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.617+0000 7fe36bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:36.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.617+0000 7fe36bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:36.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.617+0000 7fe36bfff640 10 monclient(hunting): my global_id is 34475 2024-03-25T17:35:36.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.617+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe370004eb0 con 0x7fe3840a98f0 2024-03-25T17:35:36.626 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.626+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155666133 0 0) 0x7fe3840ac910 con 0x7fe3841430c0 2024-03-25T17:35:36.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.626+0000 7fe36bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:35:36.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.626+0000 7fe36bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:36.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.626+0000 7fe36bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:36.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.626+0000 7fe36bfff640 10 monclient(hunting): my global_id is 4510 2024-03-25T17:35:36.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.626+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3840abb00 con 0x7fe3841430c0 2024-03-25T17:35:36.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.627+0000 7fe36bfff640 10 client.?.objecter ms_handle_connect 0x7fe38413f710 2024-03-25T17:35:36.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.627+0000 7fe36bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:36.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.638+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2316786820 0 0) 0x7fe3840abb00 con 0x7fe3841430c0 2024-03-25T17:35:36.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.638+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3840ac910 con 0x7fe3841430c0 2024-03-25T17:35:36.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.638+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe37c001b90 con 0x7fe3841430c0 2024-03-25T17:35:36.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.638+0000 7fe36bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.638+0000 7fe36bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.638+0000 7fe36bfff640 10 monclient(hunting): dump: 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:36.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.638+0000 7fe36bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:36.650 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3634069446 0 0) 0x7fe3840ac910 con 0x7fe3841430c0 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 >> v1:172.21.15.134:6790/0 conn(0x7fe3840a98f0 legacy=0x7fe384138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 >> v1:172.21.15.134:6789/0 conn(0x7fe38413f710 legacy=0x7fe3840ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 1 monclient: found mon.b 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3840abde0 con 0x7fe3841430c0 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 10 monclient: _finish_auth 0 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe36bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 5 monclient: authenticate success, global_id 4510 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 10 monclient: _renew_subs 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:36.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe384138f40 con 0x7fe3841430c0 2024-03-25T17:35:36.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:36.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 10 monclient: _renew_subs 2024-03-25T17:35:36.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:36.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3841363f0 con 0x7fe3841430c0 2024-03-25T17:35:36.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.649+0000 7fe38a172640 1 librados: init done 2024-03-25T17:35:36.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.650+0000 7fe38a172640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:36.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.650+0000 7fe38a172640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:36.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.650+0000 7fe38a172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:36.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.650+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3840627b0 con 0x7fe3841430c0 2024-03-25T17:35:36.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.936+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe37c002040 con 0x7fe3841430c0 2024-03-25T17:35:36.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.936+0000 7fe36bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:36.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.936+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe37c003b30 con 0x7fe3841430c0 2024-03-25T17:35:36.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.936+0000 7fe36bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:36.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.936+0000 7fe36bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.936+0000 7fe36bfff640 10 monclient: dump: 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:36.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.936+0000 7fe381ffb640 4 set_mon_vals no callback set 2024-03-25T17:35:36.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.937+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fe37c003d50 con 0x7fe3841430c0 2024-03-25T17:35:36.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.938+0000 7fe3837fe640 10 monclient: get_auth_request con 0x7fe37006de20 auth_method 0 2024-03-25T17:35:36.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.938+0000 7fe36bfff640 10 client.4510.objecter ms_handle_connect 0x7fe37006de20 2024-03-25T17:35:36.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.942+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fe37c0537e0 con 0x7fe3841430c0 2024-03-25T17:35:36.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.942+0000 7fe36bfff640 10 client.4510.objecter ms_dispatch 0x7fe384000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:36.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.942+0000 7fe36bfff640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:36.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.942+0000 7fe36bfff640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:36.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.942+0000 7fe36bfff640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T17:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.943+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe37c04f2f0 con 0x7fe3841430c0 2024-03-25T17:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.943+0000 7fe36bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:36.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:36.943+0000 7fe36bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:37.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.207+0000 7fe38a172640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:37.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.208+0000 7fe38a172640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:37.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.208+0000 7fe38a172640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:37.209 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.208+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3841434a0 con 0x7fe3841430c0 2024-03-25T17:35:37.238 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.237+0000 7fe36bfff640 1 -- 172.21.15.134:0/3793633897 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fe37c051220 con 0x7fe3841430c0 2024-03-25T17:35:37.238 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.237+0000 7fe36bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:37.238 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.237+0000 7fe36bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:37.238 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:35:37.238 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:37.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 10 librados: watch_flush enter 2024-03-25T17:35:37.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 10 librados: watch_flush exit 2024-03-25T17:35:37.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T17:35:37.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 10 client.4510.objecter successfully canceled tick 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 >> v1:172.21.15.134:6800/610837766 conn(0x7fe37006de20 legacy=0x7fe3700702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 10 monclient: shutdown 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 >> v1:172.21.15.159:6789/0 conn(0x7fe3841430c0 legacy=0x7fe3841e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 shutdown_connections 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 >> 172.21.15.134:0/3793633897 conn(0x7fe3840b7ea0 msgr2=0x7fe384074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 shutdown_connections 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 1 -- 172.21.15.134:0/3793633897 wait complete. 2024-03-25T17:35:37.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:37.240+0000 7fe38a172640 1 librados: shutdown 2024-03-25T17:35:37.254 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:44.061 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:44.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.177+0000 7f8abe9b6640 10 monclient: get_monmap_and_config 2024-03-25T17:35:44.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.177+0000 7f8abe9b6640 10 monclient: build_initial_monmap 2024-03-25T17:35:44.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.177+0000 7f8abe9b6640 10 monclient: monmap: 2024-03-25T17:35:44.178 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:44.178904+0000 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:44.178904+0000 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:44.179 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:44.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.179+0000 7f8abe9b6640 1 Processor -- start 2024-03-25T17:35:44.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.179+0000 7f8abe9b6640 1 -- start start 2024-03-25T17:35:44.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.179+0000 7f8abe9b6640 10 monclient: init 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab8142520 con 0x7f8ab813f710 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b28a0 con 0x7f8ab81430c0 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b2a30 con 0x7f8ab80a98f0 2024-03-25T17:35:44.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:44.182 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.180+0000 7f8abe9b6640 10 monclient(hunting): authenticate will time out at 5697.443231s 2024-03-25T17:35:44.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.188+0000 7f8ab7fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f8ab80a98f0 0x7f8ab80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:42592/0 (socket says 172.21.15.134:42592) 2024-03-25T17:35:44.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.188+0000 7f8ab7fff640 1 -- 172.21.15.134:0/1978413978 learned_addr learned my addr 172.21.15.134:0/1978413978 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:44.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.203+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273737870 0 0) 0x7f8ab8142520 con 0x7f8ab813f710 2024-03-25T17:35:44.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.203+0000 7f8ab77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:44.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.203+0000 7f8ab77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:44.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.203+0000 7f8ab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:44.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.203+0000 7f8ab77fe640 10 monclient(hunting): my global_id is 4491 2024-03-25T17:35:44.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.203+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a98003520 con 0x7f8ab813f710 2024-03-25T17:35:44.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.204+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821441618 0 0) 0x7f8ab80b2a30 con 0x7f8ab80a98f0 2024-03-25T17:35:44.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.204+0000 7f8ab77fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:44.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.204+0000 7f8ab77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:44.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.204+0000 7f8ab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:44.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.204+0000 7f8ab77fe640 10 monclient(hunting): my global_id is 34478 2024-03-25T17:35:44.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.204+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab8142520 con 0x7f8ab80a98f0 2024-03-25T17:35:44.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.209+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1913718035 0 0) 0x7f8a98003520 con 0x7f8ab813f710 2024-03-25T17:35:44.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 >> v1:172.21.15.134:6790/0 conn(0x7f8ab80a98f0 legacy=0x7f8ab80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:44.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 >> v1:172.21.15.159:6789/0 conn(0x7f8ab81430c0 legacy=0x7f8ab814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 10 monclient: _finish_hunting 0 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 1 monclient: found mon.noname-a 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab80b44c0 con 0x7f8ab813f710 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 10 monclient: _finish_auth 0 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 10 monclient: _check_auth_tickets 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8aa0001690 con 0x7f8ab813f710 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8ab77fe640 10 monclient: dump: 2024-03-25T17:35:44.211 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8abe9b6640 5 monclient: authenticate success, global_id 4491 2024-03-25T17:35:44.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.210+0000 7f8abe9b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8aa00017f0 con 0x7f8ab813f710 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8ab77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8ab77fe640 1 -- 172.21.15.134:0/1978413978 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8aa0001c00 con 0x7f8ab813f710 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8abe9b6640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8abe9b6640 4 set_mon_vals no callback set 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8ab77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8ab77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:44.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.212+0000 7f8ab77fe640 10 monclient: dump: 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:44.214 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:44.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.213+0000 7f8abe9b6640 10 monclient: shutdown 2024-03-25T17:35:44.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.213+0000 7f8abe9b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:44.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.213+0000 7f8abe9b6640 1 -- 172.21.15.134:0/1978413978 >> v1:172.21.15.134:6789/0 conn(0x7f8ab813f710 legacy=0x7f8ab81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:44.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.213+0000 7f8abe9b6640 1 -- 172.21.15.134:0/1978413978 shutdown_connections 2024-03-25T17:35:44.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.213+0000 7f8abe9b6640 1 -- 172.21.15.134:0/1978413978 >> 172.21.15.134:0/1978413978 conn(0x7f8ab80b7ea0 msgr2=0x7f8ab80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:44.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.213+0000 7f8abe9b6640 1 -- 172.21.15.134:0/1978413978 shutdown_connections 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.213+0000 7f8abe9b6640 1 -- 172.21.15.134:0/1978413978 wait complete. 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.214+0000 7f8abe9b6640 10 monclient: build_initial_monmap 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.214+0000 7f8abe9b6640 10 monclient: monmap: 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:44.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.214+0000 7f8abe9b6640 1 librados: starting msgr at 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.214+0000 7f8abe9b6640 1 librados: starting objecter 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.214+0000 7f8abe9b6640 1 Processor -- start 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 1 -- start start 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 1 librados: setting wanted keys 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 1 librados: calling monclient init 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient: init 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab8135070 con 0x7f8ab813f710 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b75c0 con 0x7f8ab81430c0 2024-03-25T17:35:44.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b7750 con 0x7f8ab80a98f0 2024-03-25T17:35:44.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:44.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.215+0000 7f8abe9b6640 10 monclient(hunting): authenticate will time out at 5697.478478s 2024-03-25T17:35:44.224 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.224+0000 7f8aaffff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f8ab813f710 0x7f8ab81d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:41898/0 (socket says 172.21.15.134:41898) 2024-03-25T17:35:44.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.224+0000 7f8aaffff640 1 -- 172.21.15.134:0/4182798965 learned_addr learned my addr 172.21.15.134:0/4182798965 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:44.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.225+0000 7f8ab57fa640 10 client.?.objecter ms_handle_connect 0x7f8ab813f710 2024-03-25T17:35:44.226 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.225+0000 7f8ab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:44.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.229+0000 7f8ab57fa640 10 client.?.objecter ms_handle_connect 0x7f8ab81430c0 2024-03-25T17:35:44.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.229+0000 7f8ab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:44.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.234+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2357782829 0 0) 0x7f8ab8135070 con 0x7f8ab813f710 2024-03-25T17:35:44.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.234+0000 7f8ab57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:44.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.234+0000 7f8ab57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:44.234 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.234+0000 7f8ab57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:44.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.234+0000 7f8ab57fa640 10 monclient(hunting): my global_id is 4494 2024-03-25T17:35:44.235 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.234+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c004eb0 con 0x7f8ab813f710 2024-03-25T17:35:44.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.238+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793852304 0 0) 0x7f8ab80b75c0 con 0x7f8ab81430c0 2024-03-25T17:35:44.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.238+0000 7f8ab57fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:44.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.238+0000 7f8ab57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:44.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.238+0000 7f8ab57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:44.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.238+0000 7f8ab57fa640 10 monclient(hunting): my global_id is 4516 2024-03-25T17:35:44.239 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.238+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab8135070 con 0x7f8ab81430c0 2024-03-25T17:35:44.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.247+0000 7f8ab57fa640 10 client.?.objecter ms_handle_connect 0x7f8ab80a98f0 2024-03-25T17:35:44.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.247+0000 7f8ab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:44.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.248+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3099963676 0 0) 0x7f8ab8135070 con 0x7f8ab81430c0 2024-03-25T17:35:44.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.248+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ab80b75c0 con 0x7f8ab81430c0 2024-03-25T17:35:44.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.248+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8ab0001b00 con 0x7f8ab81430c0 2024-03-25T17:35:44.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.248+0000 7f8ab57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:44.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.248+0000 7f8ab57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.248+0000 7f8ab57fa640 10 monclient(hunting): dump: 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:44.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.248+0000 7f8ab57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.255+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812022904 0 0) 0x7f8a9c004eb0 con 0x7f8ab813f710 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.255+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ab8135070 con 0x7f8ab813f710 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.255+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8aa0001c00 con 0x7f8ab813f710 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.255+0000 7f8ab57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.255+0000 7f8ab57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.255+0000 7f8ab57fa640 10 monclient(hunting): dump: 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.256 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.257 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:44.257 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:44.257 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:44.257 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:44.257 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:44.257 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2042682856 0 0) 0x7f8ab80b75c0 con 0x7f8ab81430c0 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 >> v1:172.21.15.134:6790/0 conn(0x7f8ab80a98f0 legacy=0x7f8ab81d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 >> v1:172.21.15.134:6789/0 conn(0x7f8ab813f710 legacy=0x7f8ab81d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 10 monclient: _finish_hunting 0 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 1 monclient: found mon.b 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab8135300 con 0x7f8ab81430c0 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 10 monclient: _finish_auth 0 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 10 monclient: _check_auth_tickets 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8ab57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 5 monclient: authenticate success, global_id 4516 2024-03-25T17:35:44.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 10 monclient: _renew_subs 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab8135550 con 0x7f8ab81430c0 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 10 monclient: _renew_subs 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab813b620 con 0x7f8ab81430c0 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.257+0000 7f8abe9b6640 1 librados: init done 2024-03-25T17:35:44.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.259+0000 7f8abe9b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:44.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.259+0000 7f8abe9b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:44.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.259+0000 7f8abe9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:44.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.259+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab80627b0 con 0x7f8ab81430c0 2024-03-25T17:35:44.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.545+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ab0001ff0 con 0x7f8ab81430c0 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.545+0000 7f8ab57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.545+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8ab0003b10 con 0x7f8ab81430c0 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.545+0000 7f8ab57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.545+0000 7f8ab57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.545+0000 7f8ab57fa640 10 monclient: dump: 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:44.546 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:44.547 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:44.547 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:44.547 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:44.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.545+0000 7f8ab6ffd640 4 set_mon_vals no callback set 2024-03-25T17:35:44.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.546+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f8ab0003cf0 con 0x7f8ab81430c0 2024-03-25T17:35:44.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.546+0000 7f8ab7fff640 10 monclient: get_auth_request con 0x7f8a9c06de00 auth_method 0 2024-03-25T17:35:44.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.546+0000 7f8ab57fa640 10 client.4516.objecter ms_handle_connect 0x7f8a9c06de00 2024-03-25T17:35:44.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f8ab0053840 con 0x7f8ab81430c0 2024-03-25T17:35:44.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 10 client.4516.objecter ms_dispatch 0x7f8ab8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:44.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:44.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:44.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T17:35:44.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ab004f180 con 0x7f8ab81430c0 2024-03-25T17:35:44.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:44.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.551+0000 7f8ab57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:44.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.778+0000 7f8abe9b6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:44.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.778+0000 7f8abe9b6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:44.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.778+0000 7f8abe9b6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:44.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.778+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ab80b7b10 con 0x7f8ab81430c0 2024-03-25T17:35:44.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.808+0000 7f8ab57fa640 1 -- 172.21.15.134:0/4182798965 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f8ab0051280 con 0x7f8ab81430c0 2024-03-25T17:35:44.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.808+0000 7f8ab57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:44.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.808+0000 7f8ab57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:44.809 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:35:44.809 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:44.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 10 librados: watch_flush enter 2024-03-25T17:35:44.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 10 librados: watch_flush exit 2024-03-25T17:35:44.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T17:35:44.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 10 client.4516.objecter successfully canceled tick 2024-03-25T17:35:44.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 >> v1:172.21.15.134:6800/610837766 conn(0x7f8a9c06de00 legacy=0x7f8a9c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:44.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 10 monclient: shutdown 2024-03-25T17:35:44.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:44.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 >> v1:172.21.15.159:6789/0 conn(0x7f8ab81430c0 legacy=0x7f8ab81e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:44.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 shutdown_connections 2024-03-25T17:35:44.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 >> 172.21.15.134:0/4182798965 conn(0x7f8ab80b7ea0 msgr2=0x7f8ab8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:44.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 shutdown_connections 2024-03-25T17:35:44.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 1 -- 172.21.15.134:0/4182798965 wait complete. 2024-03-25T17:35:44.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:44.811+0000 7f8abe9b6640 1 librados: shutdown 2024-03-25T17:35:44.825 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:50.170 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:55.186 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.176 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:55.192 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.201 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:06:55.217 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.222 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:55.251 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.224 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:55.240 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.232 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:55.261 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.239 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:55.255 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.265 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:06:55.294 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:50.272 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:06:55.289 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:35:51.630 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:51.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.745+0000 7fe3cac0f640 10 monclient: get_monmap_and_config 2024-03-25T17:35:51.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.745+0000 7fe3cac0f640 10 monclient: build_initial_monmap 2024-03-25T17:35:51.746 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.745+0000 7fe3cac0f640 10 monclient: monmap: 2024-03-25T17:35:51.746 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:51.746 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:51.746888+0000 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:51.746888+0000 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:51.747 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:51.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.746+0000 7fe3cac0f640 1 Processor -- start 2024-03-25T17:35:51.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.746+0000 7fe3cac0f640 1 -- start start 2024-03-25T17:35:51.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.746+0000 7fe3cac0f640 10 monclient: init 2024-03-25T17:35:51.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c40a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c41430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c4142520 con 0x7fe3c41430c0 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40b28a0 con 0x7fe3c413f710 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c40b2a30 con 0x7fe3c40a98f0 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:51.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.747+0000 7fe3cac0f640 10 monclient(hunting): authenticate will time out at 5705.010492s 2024-03-25T17:35:51.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.756+0000 7fe3c89ab640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe3c40a98f0 0x7fe3c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59430/0 (socket says 172.21.15.134:59430) 2024-03-25T17:35:51.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.756+0000 7fe3c89ab640 1 -- 172.21.15.134:0/1025594749 learned_addr learned my addr 172.21.15.134:0/1025594749 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:51.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.768+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43726792 0 0) 0x7fe3c40b2a30 con 0x7fe3c40a98f0 2024-03-25T17:35:51.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.768+0000 7fe3c37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:51.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.768+0000 7fe3c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:51.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.768+0000 7fe3c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:51.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.768+0000 7fe3c37fe640 10 monclient(hunting): my global_id is 34484 2024-03-25T17:35:51.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.768+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b4003520 con 0x7fe3c40a98f0 2024-03-25T17:35:51.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.787+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1600794608 0 0) 0x7fe3c40b28a0 con 0x7fe3c413f710 2024-03-25T17:35:51.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.787+0000 7fe3c37fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:51.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.787+0000 7fe3c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:51.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.787+0000 7fe3c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:51.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.787+0000 7fe3c37fe640 10 monclient(hunting): my global_id is 4519 2024-03-25T17:35:51.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.787+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c40b2a30 con 0x7fe3c413f710 2024-03-25T17:35:51.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2987403653 0 0) 0x7fe3b4003520 con 0x7fe3c40a98f0 2024-03-25T17:35:51.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 >> v1:172.21.15.159:6789/0 conn(0x7fe3c413f710 legacy=0x7fe3c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:51.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 >> v1:172.21.15.134:6789/0 conn(0x7fe3c41430c0 legacy=0x7fe3c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:51.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 10 monclient: _finish_hunting 0 2024-03-25T17:35:51.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 1 monclient: found mon.noname-b 2024-03-25T17:35:51.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:51.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c40b44c0 con 0x7fe3c40a98f0 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 10 monclient: _finish_auth 0 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 10 monclient: _check_auth_tickets 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe3b8001690 con 0x7fe3c40a98f0 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3cac0f640 5 monclient: authenticate success, global_id 34484 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3cac0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3c37fe640 10 monclient: dump: 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:51.802 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:51.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.800+0000 7fe3cac0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:51.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3b80017f0 con 0x7fe3c40a98f0 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3c37fe640 1 -- 172.21.15.134:0/1025594749 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe3b8001c00 con 0x7fe3c40a98f0 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3c37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3c37fe640 10 monclient: dump: 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:51.810 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:51.811 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:51.811 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:51.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3cac0f640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:51.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.809+0000 7fe3cac0f640 4 set_mon_vals no callback set 2024-03-25T17:35:51.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.810+0000 7fe3cac0f640 10 monclient: shutdown 2024-03-25T17:35:51.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.810+0000 7fe3cac0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:51.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.810+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1025594749 >> v1:172.21.15.134:6790/0 conn(0x7fe3c40a98f0 legacy=0x7fe3c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:51.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.810+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1025594749 shutdown_connections 2024-03-25T17:35:51.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.810+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1025594749 >> 172.21.15.134:0/1025594749 conn(0x7fe3c40b7ea0 msgr2=0x7fe3c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:51.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.810+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1025594749 shutdown_connections 2024-03-25T17:35:51.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.810+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1025594749 wait complete. 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 10 monclient: build_initial_monmap 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 10 monclient: monmap: 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 1 librados: starting msgr at 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 1 librados: starting objecter 2024-03-25T17:35:51.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 1 Processor -- start 2024-03-25T17:35:51.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 1 -- start start 2024-03-25T17:35:51.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 1 librados: setting wanted keys 2024-03-25T17:35:51.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 1 librados: calling monclient init 2024-03-25T17:35:51.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.811+0000 7fe3cac0f640 10 monclient: init 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.812+0000 7fe3cac0f640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.812+0000 7fe3cac0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.812+0000 7fe3cac0f640 10 monclient(hunting): picked mon.noname-b con 0x7fe3c40a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.812+0000 7fe3cac0f640 10 monclient(hunting): picked mon.noname-c con 0x7fe3c413f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.813+0000 7fe3cac0f640 10 monclient(hunting): picked mon.noname-a con 0x7fe3c41430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.813+0000 7fe3cac0f640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c4138b20 con 0x7fe3c41430c0 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.813+0000 7fe3cac0f640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c4137a80 con 0x7fe3c40a98f0 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.813+0000 7fe3cac0f640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c4139880 con 0x7fe3c413f710 2024-03-25T17:35:51.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.813+0000 7fe3cac0f640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:51.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.813+0000 7fe3cac0f640 10 monclient(hunting): authenticate will time out at 5705.076007s 2024-03-25T17:35:51.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.815+0000 7fe3c3fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fe3c413f710 0x7fe3c4139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:59460/0 (socket says 172.21.15.134:59460) 2024-03-25T17:35:51.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.815+0000 7fe3c3fff640 1 -- 172.21.15.134:0/1058415804 learned_addr learned my addr 172.21.15.134:0/1058415804 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:51.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.822+0000 7fe3c17fa640 10 client.?.objecter ms_handle_connect 0x7fe3c413f710 2024-03-25T17:35:51.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.822+0000 7fe3c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:51.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.831+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229736562 0 0) 0x7fe3c4139880 con 0x7fe3c413f710 2024-03-25T17:35:51.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.831+0000 7fe3c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:51.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.831+0000 7fe3c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:51.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.831+0000 7fe3c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:51.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.831+0000 7fe3c17fa640 10 monclient(hunting): my global_id is 34487 2024-03-25T17:35:51.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.831+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a8004eb0 con 0x7fe3c413f710 2024-03-25T17:35:51.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.838+0000 7fe3c17fa640 10 client.?.objecter ms_handle_connect 0x7fe3c40a98f0 2024-03-25T17:35:51.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.838+0000 7fe3c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:51.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.847+0000 7fe3c17fa640 10 client.?.objecter ms_handle_connect 0x7fe3c41430c0 2024-03-25T17:35:51.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.847+0000 7fe3c17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:51.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.856+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311392328 0 0) 0x7fe3c4137a80 con 0x7fe3c40a98f0 2024-03-25T17:35:51.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.856+0000 7fe3c17fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:51.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.856+0000 7fe3c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:51.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.856+0000 7fe3c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:51.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.856+0000 7fe3c17fa640 10 monclient(hunting): my global_id is 4522 2024-03-25T17:35:51.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.856+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3c4139880 con 0x7fe3c40a98f0 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.858+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1165002099 0 0) 0x7fe3a8004eb0 con 0x7fe3c413f710 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.858+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3c4137a80 con 0x7fe3c413f710 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.858+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe3b0001a10 con 0x7fe3c413f710 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.858+0000 7fe3c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.858+0000 7fe3c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.858+0000 7fe3c17fa640 10 monclient(hunting): dump: 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:51.859 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:51.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.858+0000 7fe3c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3514876407 0 0) 0x7fe3c4137a80 con 0x7fe3c413f710 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 >> v1:172.21.15.159:6789/0 conn(0x7fe3c40a98f0 legacy=0x7fe3c4137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 >> v1:172.21.15.134:6789/0 conn(0x7fe3c41430c0 legacy=0x7fe3c41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 10 monclient: _finish_hunting 0 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 1 monclient: found mon.c 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4139a10 con 0x7fe3c413f710 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 10 monclient: _finish_auth 0 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 10 monclient: _check_auth_tickets 2024-03-25T17:35:51.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 5 monclient: authenticate success, global_id 34487 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 10 monclient: _renew_subs 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c41dab00 con 0x7fe3c413f710 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 10 client.34487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 10 monclient: _renew_subs 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c413b4f0 con 0x7fe3c413f710 2024-03-25T17:35:51.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.867+0000 7fe3cac0f640 1 librados: init done 2024-03-25T17:35:51.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.868+0000 7fe3cac0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:51.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.868+0000 7fe3cac0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:51.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.868+0000 7fe3cac0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:51.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:51.868+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c41dae00 con 0x7fe3c413f710 2024-03-25T17:35:52.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.165+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3b0001c50 con 0x7fe3c413f710 2024-03-25T17:35:52.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.165+0000 7fe3c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:52.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.165+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe3b0003a60 con 0x7fe3c413f710 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.165+0000 7fe3c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.165+0000 7fe3c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.165+0000 7fe3c17fa640 10 monclient: dump: 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:52.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.165+0000 7fe3c2ffd640 4 set_mon_vals no callback set 2024-03-25T17:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.166+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fe3b0003c80 con 0x7fe3c413f710 2024-03-25T17:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.166+0000 7fe3c89ab640 10 monclient: get_auth_request con 0x7fe3a806dda0 auth_method 0 2024-03-25T17:35:52.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.167+0000 7fe3c17fa640 10 client.34487.objecter ms_handle_connect 0x7fe3a806dda0 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fe3b0054870 con 0x7fe3c413f710 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 10 client.34487.objecter ms_dispatch 0x7fe3c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 3 client.34487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 3 client.34487.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 20 client.34487.objecter dump_active .. 0 homeless 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3b0050380 con 0x7fe3c413f710 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:52.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.172+0000 7fe3c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:35:52.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.423+0000 7fe3cac0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:52.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.423+0000 7fe3cac0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:52.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.423+0000 7fe3cac0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:35:52.424 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.423+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c40b3e80 con 0x7fe3c413f710 2024-03-25T17:35:52.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.597+0000 7fe3c17fa640 1 -- 172.21.15.134:0/1058415804 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fe3b00522b0 con 0x7fe3c413f710 2024-03-25T17:35:52.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.597+0000 7fe3c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:35:52.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.597+0000 7fe3c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:35:52.599 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:35:52.599 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:35:52.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 10 librados: watch_flush enter 2024-03-25T17:35:52.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 10 librados: watch_flush exit 2024-03-25T17:35:52.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 20 client.34487.objecter shutdown clearing up homeless session... 2024-03-25T17:35:52.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 10 client.34487.objecter successfully canceled tick 2024-03-25T17:35:52.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 >> v1:172.21.15.134:6800/610837766 conn(0x7fe3a806dda0 legacy=0x7fe3a8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:52.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 10 monclient: shutdown 2024-03-25T17:35:52.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:52.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 >> v1:172.21.15.134:6790/0 conn(0x7fe3c413f710 legacy=0x7fe3c4139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:52.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 shutdown_connections 2024-03-25T17:35:52.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 >> 172.21.15.134:0/1058415804 conn(0x7fe3c40b7ea0 msgr2=0x7fe3c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:52.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 shutdown_connections 2024-03-25T17:35:52.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.600+0000 7fe3cac0f640 1 -- 172.21.15.134:0/1058415804 wait complete. 2024-03-25T17:35:52.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:52.601+0000 7fe3cac0f640 1 librados: shutdown 2024-03-25T17:35:52.613 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:35:59.420 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:35:59.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.536+0000 7f9fc2bc8640 10 monclient: get_monmap_and_config 2024-03-25T17:35:59.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.536+0000 7f9fc2bc8640 10 monclient: build_initial_monmap 2024-03-25T17:35:59.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.536+0000 7f9fc2bc8640 10 monclient: monmap: 2024-03-25T17:35:59.537 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:59.537 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:35:59.537701+0000 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:35:59.537701+0000 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:35:59.538 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.537+0000 7f9fc2bc8640 1 Processor -- start 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.537+0000 7f9fc2bc8640 1 -- start start 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.537+0000 7f9fc2bc8640 10 monclient: init 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fbc0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fbc13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fbc1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc142520 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0b28a0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0b2a30 con 0x7f9fbc13f710 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:59.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.538+0000 7f9fc2bc8640 10 monclient(hunting): authenticate will time out at 5712.801056s 2024-03-25T17:35:59.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.551+0000 7f9fbbfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f9fbc13f710 0x7f9fbc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50166/0 (socket says 172.21.15.134:50166) 2024-03-25T17:35:59.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.552+0000 7f9fbbfff640 1 -- 172.21.15.134:0/2038523934 learned_addr learned my addr 172.21.15.134:0/2038523934 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:59.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.562+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 512214736 0 0) 0x7f9fbc0b28a0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.562+0000 7f9fbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:59.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.562+0000 7f9fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:59.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.562+0000 7f9fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:59.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.562+0000 7f9fbb7fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T17:35:59.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.562+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fac003520 con 0x7f9fbc1430c0 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123726458 0 0) 0x7f9fbc0b2a30 con 0x7f9fbc13f710 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): my global_id is 34490 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fbc0b28a0 con 0x7f9fbc13f710 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991910915 0 0) 0x7f9fbc142520 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:59.567 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 10 monclient(hunting): my global_id is 4503 2024-03-25T17:35:59.568 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.566+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fbc0b2a30 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.572+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2703321539 0 0) 0x7f9fac003520 con 0x7f9fbc1430c0 2024-03-25T17:35:59.573 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.572+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 >> v1:172.21.15.134:6790/0 conn(0x7f9fbc13f710 legacy=0x7f9fbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 >> v1:172.21.15.134:6789/0 conn(0x7f9fbc0a98f0 legacy=0x7f9fbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 1 monclient: found mon.noname-c 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0b44c0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 10 monclient: _finish_auth 0 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9fb4001a10 con 0x7f9fbc1430c0 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:59.574 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fbb7fe640 10 monclient: dump: 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fc2bc8640 5 monclient: authenticate success, global_id 4525 2024-03-25T17:35:59.575 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.573+0000 7f9fc2bc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:35:59.576 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fb4001b70 con 0x7f9fbc1430c0 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fbb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fbb7fe640 1 -- 172.21.15.134:0/2038523934 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9fb4001f80 con 0x7f9fbc1430c0 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fbb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fbb7fe640 10 monclient: dump: 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.577 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:59.578 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:59.578 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:59.578 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:59.578 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:59.578 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fc2bc8640 10 monclient: get_monmap_and_config success 2024-03-25T17:35:59.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.576+0000 7f9fc2bc8640 4 set_mon_vals no callback set 2024-03-25T17:35:59.579 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.577+0000 7f9fc2bc8640 10 monclient: shutdown 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.577+0000 7f9fc2bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.577+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/2038523934 >> v1:172.21.15.159:6789/0 conn(0x7f9fbc1430c0 legacy=0x7f9fbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.577+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/2038523934 shutdown_connections 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.577+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/2038523934 >> 172.21.15.134:0/2038523934 conn(0x7f9fbc0b7ea0 msgr2=0x7f9fbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.577+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/2038523934 shutdown_connections 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.577+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/2038523934 wait complete. 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 10 monclient: build_initial_monmap 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 10 monclient: monmap: 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:35:59.580 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 1 librados: starting msgr at 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 1 librados: starting objecter 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 1 Processor -- start 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 1 -- start start 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 1 librados: setting wanted keys 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 1 librados: calling monclient init 2024-03-25T17:35:59.581 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.578+0000 7f9fc2bc8640 10 monclient: init 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.579+0000 7f9fc2bc8640 10 monclient: _reopen_session rank -1 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.579+0000 7f9fc2bc8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.579+0000 7f9fc2bc8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fbc0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.579+0000 7f9fc2bc8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fbc13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.579+0000 7f9fc2bc8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fbc1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.580+0000 7f9fc2bc8640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc1daca0 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.580+0000 7f9fc2bc8640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc138ab0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.580+0000 7f9fc2bc8640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0aba90 con 0x7f9fbc13f710 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.580+0000 7f9fc2bc8640 10 monclient(hunting): _renew_subs 2024-03-25T17:35:59.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.580+0000 7f9fc2bc8640 10 monclient(hunting): authenticate will time out at 5712.842861s 2024-03-25T17:35:59.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.586+0000 7f9fbbfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f9fbc13f710 0x7f9fbc1db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50192/0 (socket says 172.21.15.134:50192) 2024-03-25T17:35:59.586 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.586+0000 7f9fbbfff640 1 -- 172.21.15.134:0/1424038058 learned_addr learned my addr 172.21.15.134:0/1424038058 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:35:59.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.590+0000 7f9fb97fa640 10 client.?.objecter ms_handle_connect 0x7f9fbc1430c0 2024-03-25T17:35:59.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.590+0000 7f9fb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:59.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.590+0000 7f9fb97fa640 10 client.?.objecter ms_handle_connect 0x7f9fbc13f710 2024-03-25T17:35:59.591 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.590+0000 7f9fb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:59.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.596+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147376 0 0) 0x7f9fbc138ab0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.596+0000 7f9fb97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:59.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.596+0000 7f9fb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:59.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.596+0000 7f9fb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:59.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.596+0000 7f9fb97fa640 10 monclient(hunting): my global_id is 4528 2024-03-25T17:35:59.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.596+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa0004eb0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.598+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3694150598 0 0) 0x7f9fbc0aba90 con 0x7f9fbc13f710 2024-03-25T17:35:59.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.598+0000 7f9fb97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:59.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.598+0000 7f9fb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:59.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.598+0000 7f9fb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:59.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.598+0000 7f9fb97fa640 10 monclient(hunting): my global_id is 34493 2024-03-25T17:35:59.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.598+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fbc138ab0 con 0x7f9fbc13f710 2024-03-25T17:35:59.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.600+0000 7f9fb97fa640 10 client.?.objecter ms_handle_connect 0x7f9fbc0a98f0 2024-03-25T17:35:59.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.600+0000 7f9fb97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:35:59.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.607+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426428158 0 0) 0x7f9fbc1daca0 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.607+0000 7f9fb97fa640 10 monclient(hunting): _negotiate 2024-03-25T17:35:59.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.607+0000 7f9fb97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:35:59.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.607+0000 7f9fb97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:35:59.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.607+0000 7f9fb97fa640 10 monclient(hunting): my global_id is 4506 2024-03-25T17:35:59.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.607+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fbc0aba90 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.613+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4065226967 0 0) 0x7f9fa0004eb0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.613+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fbc1daca0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.613+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9fb4001e80 con 0x7f9fbc1430c0 2024-03-25T17:35:59.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.613+0000 7f9fb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:59.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.613+0000 7f9fb97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.613+0000 7f9fb97fa640 10 monclient(hunting): dump: 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.613+0000 7f9fb97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:35:59.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.621+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1346345905 0 0) 0x7f9fbc0aba90 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.621+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa0004eb0 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.622 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.621+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9fb00018b0 con 0x7f9fbc0a98f0 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.622+0000 7f9fb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.622+0000 7f9fb97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.622+0000 7f9fb97fa640 10 monclient(hunting): dump: 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:59.623 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.624+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688834590 0 0) 0x7f9fbc138ab0 con 0x7f9fbc13f710 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.624+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fbc0aba90 con 0x7f9fbc13f710 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.624+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9fa80019b0 con 0x7f9fbc13f710 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.624+0000 7f9fb97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.624+0000 7f9fb97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.624+0000 7f9fb97fa640 10 monclient(hunting): dump: 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:59.625 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:59.626 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:59.626 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:59.626 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:59.626 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3818876888 0 0) 0x7f9fbc1daca0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 >> v1:172.21.15.134:6790/0 conn(0x7f9fbc13f710 legacy=0x7f9fbc1db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:59.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 >> v1:172.21.15.134:6789/0 conn(0x7f9fbc0a98f0 legacy=0x7f9fbc1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:35:59.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 10 monclient: _finish_hunting 0 2024-03-25T17:35:59.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 1 monclient: found mon.b 2024-03-25T17:35:59.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:35:59.630 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc1d41e0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 10 monclient: _finish_auth 0 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 10 monclient: _check_auth_tickets 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fb97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 5 monclient: authenticate success, global_id 4528 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 10 monclient: _renew_subs 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc1db840 con 0x7f9fbc1430c0 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 10 monclient: _renew_subs 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc138f00 con 0x7f9fbc1430c0 2024-03-25T17:35:59.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.629+0000 7f9fc2bc8640 1 librados: init done 2024-03-25T17:35:59.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.630+0000 7f9fc2bc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:59.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.630+0000 7f9fc2bc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:59.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.630+0000 7f9fc2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:35:59.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.630+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc1dae50 con 0x7f9fbc1430c0 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.913+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fb4002020 con 0x7f9fbc1430c0 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.913+0000 7f9fb97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.913+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9fb4003c20 con 0x7f9fbc1430c0 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.913+0000 7f9fb97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.913+0000 7f9fb97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.913+0000 7f9fb97fa640 10 monclient: dump: 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:35:59.915 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:35:59.916 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:35:59.916 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:35:59.916 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:35:59.916 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:35:59.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.913+0000 7f9fbaffd640 4 set_mon_vals no callback set 2024-03-25T17:35:59.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.914+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f9fb4019bf0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.914+0000 7f9fc0964640 10 monclient: get_auth_request con 0x7f9fa006e010 auth_method 0 2024-03-25T17:35:59.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.915+0000 7f9fb97fa640 10 client.4528.objecter ms_handle_connect 0x7f9fa006e010 2024-03-25T17:35:59.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.919+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f9fb4053780 con 0x7f9fbc1430c0 2024-03-25T17:35:59.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.919+0000 7f9fb97fa640 10 client.4528.objecter ms_dispatch 0x7f9fbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:35:59.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.919+0000 7f9fb97fa640 3 client.4528.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:35:59.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.919+0000 7f9fb97fa640 3 client.4528.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:35:59.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.919+0000 7f9fb97fa640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-25T17:35:59.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.920+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fb404f0c0 con 0x7f9fbc1430c0 2024-03-25T17:35:59.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.920+0000 7f9fb97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:35:59.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:35:59.920+0000 7f9fb97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:00.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.141+0000 7f9fc2bc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:00.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.141+0000 7f9fc2bc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:00.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.141+0000 7f9fc2bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:00.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.141+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fbc0627b0 con 0x7f9fbc1430c0 2024-03-25T17:36:00.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.165+0000 7f9fb97fa640 1 -- 172.21.15.134:0/1424038058 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f9fb40511c0 con 0x7f9fbc1430c0 2024-03-25T17:36:00.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.165+0000 7f9fb97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:00.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.165+0000 7f9fb97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:00.166 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:00.167 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:00.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 10 librados: watch_flush enter 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 10 librados: watch_flush exit 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 10 client.4528.objecter successfully canceled tick 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 >> v1:172.21.15.134:6800/610837766 conn(0x7f9fa006e010 legacy=0x7f9fa00704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 10 monclient: shutdown 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 >> v1:172.21.15.159:6789/0 conn(0x7f9fbc1430c0 legacy=0x7f9fbc1d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 shutdown_connections 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 >> 172.21.15.134:0/1424038058 conn(0x7f9fbc0b7ea0 msgr2=0x7f9fbc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 shutdown_connections 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.168+0000 7f9fc2bc8640 1 -- 172.21.15.134:0/1424038058 wait complete. 2024-03-25T17:36:00.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:00.169+0000 7f9fc2bc8640 1 librados: shutdown 2024-03-25T17:36:00.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:06.987 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.102+0000 7fccb1e8d640 10 monclient: get_monmap_and_config 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.102+0000 7fccb1e8d640 10 monclient: build_initial_monmap 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.102+0000 7fccb1e8d640 10 monclient: monmap: 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:36:07.103354+0000 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:36:07.103354+0000 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:36:07.103 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:36:07.104 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.103+0000 7fccb1e8d640 1 Processor -- start 2024-03-25T17:36:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.103+0000 7fccb1e8d640 1 -- start start 2024-03-25T17:36:07.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.103+0000 7fccb1e8d640 10 monclient: init 2024-03-25T17:36:07.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:07.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:36:07.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 10 monclient(hunting): picked mon.noname-b con 0x7fccac06c380 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:07.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 10 monclient(hunting): picked mon.noname-a con 0x7fccac0b12d0 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:07.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 10 monclient(hunting): picked mon.noname-c con 0x7fccac0aedf0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:07.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0b16f0 con 0x7fccac0b12d0 2024-03-25T17:36:07.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0adfe0 con 0x7fccac0aedf0 2024-03-25T17:36:07.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac0add00 con 0x7fccac06c380 2024-03-25T17:36:07.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:07.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.104+0000 7fccb1e8d640 10 monclient(hunting): authenticate will time out at 5720.367337s 2024-03-25T17:36:07.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.115+0000 7fccabfff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fccac0aedf0 0x7fccac14e050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:44122/0 (socket says 172.21.15.134:44122) 2024-03-25T17:36:07.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.115+0000 7fccabfff640 1 -- 172.21.15.134:0/2013024227 learned_addr learned my addr 172.21.15.134:0/2013024227 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:07.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.123+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1878907592 0 0) 0x7fccac0adfe0 con 0x7fccac0aedf0 2024-03-25T17:36:07.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.123+0000 7fccaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:07.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.123+0000 7fccaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:07.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.123+0000 7fccaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:07.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.123+0000 7fccaa7fc640 10 monclient(hunting): my global_id is 4531 2024-03-25T17:36:07.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.123+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc94003520 con 0x7fccac0aedf0 2024-03-25T17:36:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.132+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946014229 0 0) 0x7fccac0b16f0 con 0x7fccac0b12d0 2024-03-25T17:36:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.132+0000 7fccaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.132+0000 7fccaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.132+0000 7fccaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.132+0000 7fccaa7fc640 10 monclient(hunting): my global_id is 4509 2024-03-25T17:36:07.133 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.132+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac0adfe0 con 0x7fccac0b12d0 2024-03-25T17:36:07.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.139+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225144110 0 0) 0x7fccac0add00 con 0x7fccac06c380 2024-03-25T17:36:07.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.139+0000 7fccaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:07.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.139+0000 7fccaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:07.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.139+0000 7fccaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:07.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.139+0000 7fccaa7fc640 10 monclient(hunting): my global_id is 34496 2024-03-25T17:36:07.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.139+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac0b16f0 con 0x7fccac06c380 2024-03-25T17:36:07.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3637561451 0 0) 0x7fcc94003520 con 0x7fccac0aedf0 2024-03-25T17:36:07.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 >> v1:172.21.15.134:6790/0 conn(0x7fccac06c380 legacy=0x7fccac0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 >> v1:172.21.15.134:6789/0 conn(0x7fccac0b12d0 legacy=0x7fccac0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 1 monclient: found mon.noname-c 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac0ae2c0 con 0x7fccac0aedf0 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 10 monclient: _finish_auth 0 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcca4001a10 con 0x7fccac0aedf0 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.143+0000 7fccaa7fc640 10 monclient: dump: 2024-03-25T17:36:07.145 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.144+0000 7fccb1e8d640 5 monclient: authenticate success, global_id 4531 2024-03-25T17:36:07.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.144+0000 7fccb1e8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:07.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcca4001b70 con 0x7fccac0aedf0 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccaa7fc640 1 -- 172.21.15.134:0/2013024227 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcca4001f00 con 0x7fccac0aedf0 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccb1e8d640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccb1e8d640 4 set_mon_vals no callback set 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.151+0000 7fccaa7fc640 10 monclient: dump: 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:07.152 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:07.153 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:07.153 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 10 monclient: shutdown 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 1 -- 172.21.15.134:0/2013024227 >> v1:172.21.15.159:6789/0 conn(0x7fccac0aedf0 legacy=0x7fccac14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 1 -- 172.21.15.134:0/2013024227 shutdown_connections 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 1 -- 172.21.15.134:0/2013024227 >> 172.21.15.134:0/2013024227 conn(0x7fccac0b5d10 msgr2=0x7fccac0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 1 -- 172.21.15.134:0/2013024227 shutdown_connections 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 1 -- 172.21.15.134:0/2013024227 wait complete. 2024-03-25T17:36:07.154 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 10 monclient: build_initial_monmap 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.152+0000 7fccb1e8d640 10 monclient: monmap: 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.153+0000 7fccb1e8d640 1 librados: starting msgr at 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.153+0000 7fccb1e8d640 1 librados: starting objecter 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.153+0000 7fccb1e8d640 1 Processor -- start 2024-03-25T17:36:07.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.153+0000 7fccb1e8d640 1 -- start start 2024-03-25T17:36:07.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.153+0000 7fccb1e8d640 1 librados: setting wanted keys 2024-03-25T17:36:07.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.153+0000 7fccb1e8d640 1 librados: calling monclient init 2024-03-25T17:36:07.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.153+0000 7fccb1e8d640 10 monclient: init 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 10 monclient(hunting): picked mon.noname-b con 0x7fccac06c380 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 10 monclient(hunting): picked mon.noname-a con 0x7fccac0aedf0 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 10 monclient(hunting): picked mon.noname-c con 0x7fccac0b12d0 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac06c8e0 con 0x7fccac0aedf0 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac1a4150 con 0x7fccac06c380 2024-03-25T17:36:07.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccac1c71a0 con 0x7fccac0b12d0 2024-03-25T17:36:07.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:07.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.154+0000 7fccb1e8d640 10 monclient(hunting): authenticate will time out at 5720.417201s 2024-03-25T17:36:07.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.170+0000 7fccabfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fccac0b12d0 0x7fccac1d7f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:54764/0 (socket says 172.21.15.134:54764) 2024-03-25T17:36:07.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.170+0000 7fccabfff640 1 -- 172.21.15.134:0/156099487 learned_addr learned my addr 172.21.15.134:0/156099487 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:07.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.172+0000 7fcc93fff640 10 client.?.objecter ms_handle_connect 0x7fccac0b12d0 2024-03-25T17:36:07.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.172+0000 7fcc93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:07.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.173+0000 7fcc93fff640 10 client.?.objecter ms_handle_connect 0x7fccac0aedf0 2024-03-25T17:36:07.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.173+0000 7fcc93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:07.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.187+0000 7fcc93fff640 10 client.?.objecter ms_handle_connect 0x7fccac06c380 2024-03-25T17:36:07.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.187+0000 7fcc93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:07.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.190+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706313443 0 0) 0x7fccac1c71a0 con 0x7fccac0b12d0 2024-03-25T17:36:07.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.190+0000 7fcc93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:07.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.190+0000 7fcc93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:07.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.190+0000 7fcc93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:07.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.190+0000 7fcc93fff640 10 monclient(hunting): my global_id is 34499 2024-03-25T17:36:07.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.190+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc98004eb0 con 0x7fccac0b12d0 2024-03-25T17:36:07.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.197+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3677233653 0 0) 0x7fccac06c8e0 con 0x7fccac0aedf0 2024-03-25T17:36:07.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.197+0000 7fcc93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:07.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.197+0000 7fcc93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:07.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.197+0000 7fcc93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:07.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.197+0000 7fcc93fff640 10 monclient(hunting): my global_id is 4512 2024-03-25T17:36:07.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.197+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac1c71a0 con 0x7fccac0aedf0 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.211+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 525135633 0 0) 0x7fcc98004eb0 con 0x7fccac0b12d0 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.211+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccac06c8e0 con 0x7fccac0b12d0 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.211+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcca4001e80 con 0x7fccac0b12d0 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.211+0000 7fcc93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.211+0000 7fcc93fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.211+0000 7fcc93fff640 10 monclient(hunting): dump: 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:07.212 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.211+0000 7fcc93fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.212+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972407090 0 0) 0x7fccac1a4150 con 0x7fccac06c380 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.212+0000 7fcc93fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.212+0000 7fcc93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.212+0000 7fcc93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:07.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.212+0000 7fcc93fff640 10 monclient(hunting): my global_id is 4534 2024-03-25T17:36:07.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.212+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc98004eb0 con 0x7fccac06c380 2024-03-25T17:36:07.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1665211023 0 0) 0x7fccac06c8e0 con 0x7fccac0b12d0 2024-03-25T17:36:07.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 >> v1:172.21.15.159:6789/0 conn(0x7fccac06c380 legacy=0x7fccac1a2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:07.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 >> v1:172.21.15.134:6789/0 conn(0x7fccac0aedf0 legacy=0x7fccac1d7850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:07.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 10 monclient: _finish_hunting 0 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 1 monclient: found mon.c 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccac1a3d40 con 0x7fccac0b12d0 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 10 monclient: _finish_auth 0 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 10 monclient: _check_auth_tickets 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fcc93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 5 monclient: authenticate success, global_id 34499 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 10 monclient: _renew_subs 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccac1c4ea0 con 0x7fccac0b12d0 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 10 client.34499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 10 monclient: _renew_subs 2024-03-25T17:36:07.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:07.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccac1e3320 con 0x7fccac0b12d0 2024-03-25T17:36:07.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.229+0000 7fccb1e8d640 1 librados: init done 2024-03-25T17:36:07.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.230+0000 7fccb1e8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:07.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.230+0000 7fccb1e8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:07.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.230+0000 7fccb1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:07.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.230+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccac0627b0 con 0x7fccac0b12d0 2024-03-25T17:36:07.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.553+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcca4004e10 con 0x7fccac0b12d0 2024-03-25T17:36:07.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.553+0000 7fcc93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:07.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.553+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcca4005120 con 0x7fccac0b12d0 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.553+0000 7fcc93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.553+0000 7fcc93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.553+0000 7fcc93fff640 10 monclient: dump: 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:07.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.553+0000 7fcca9ffb640 4 set_mon_vals no callback set 2024-03-25T17:36:07.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.554+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fcca401b0f0 con 0x7fccac0b12d0 2024-03-25T17:36:07.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.555+0000 7fccab7fe640 10 monclient: get_auth_request con 0x7fcc9806ddb0 auth_method 0 2024-03-25T17:36:07.556 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.555+0000 7fcc93fff640 10 client.34499.objecter ms_handle_connect 0x7fcc9806ddb0 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fcca4054be0 con 0x7fccac0b12d0 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 10 client.34499.objecter ms_dispatch 0x7fccac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 3 client.34499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 3 client.34499.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 20 client.34499.objecter dump_active .. 0 homeless 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcca40506f0 con 0x7fccac0b12d0 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:07.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.560+0000 7fcc93fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:07.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.784+0000 7fccb1e8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:07.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.784+0000 7fccb1e8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:07.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.784+0000 7fccb1e8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:07.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.784+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccac06d790 con 0x7fccac0b12d0 2024-03-25T17:36:07.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.821+0000 7fcc93fff640 1 -- 172.21.15.134:0/156099487 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fccac06d790 con 0x7fccac0b12d0 2024-03-25T17:36:07.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.821+0000 7fcc93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:07.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.821+0000 7fcc93fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:07.823 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:07.823 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 10 librados: watch_flush enter 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 10 librados: watch_flush exit 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 20 client.34499.objecter shutdown clearing up homeless session... 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 10 client.34499.objecter successfully canceled tick 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 >> v1:172.21.15.134:6800/610837766 conn(0x7fcc9806ddb0 legacy=0x7fcc98070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 10 monclient: shutdown 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 >> v1:172.21.15.134:6790/0 conn(0x7fccac0b12d0 legacy=0x7fccac1d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 shutdown_connections 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.824+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 >> 172.21.15.134:0/156099487 conn(0x7fccac0b5d10 msgr2=0x7fccac0ae6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:07.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.825+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 shutdown_connections 2024-03-25T17:36:07.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.825+0000 7fccb1e8d640 1 -- 172.21.15.134:0/156099487 wait complete. 2024-03-25T17:36:07.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:07.825+0000 7fccb1e8d640 1 librados: shutdown 2024-03-25T17:36:07.837 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:10.171 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:15.187 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.220 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:15.236 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.247 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:15.264 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.262 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:15.291 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.273 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:15.302 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.293 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:15.310 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.311 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:15.327 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.321 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:15.348 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:10.342 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:15.359 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:14.644 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:14.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.760+0000 7fee71405640 10 monclient: get_monmap_and_config 2024-03-25T17:36:14.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.760+0000 7fee71405640 10 monclient: build_initial_monmap 2024-03-25T17:36:14.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.760+0000 7fee71405640 10 monclient: monmap: 2024-03-25T17:36:14.761 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:14.761 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:14.761 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:36:14.761698+0000 2024-03-25T17:36:14.761 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:36:14.761698+0000 2024-03-25T17:36:14.762 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:14.762 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:14.762 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:14.762 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:36:14.762 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:36:14.762 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:14.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.761+0000 7fee71405640 1 Processor -- start 2024-03-25T17:36:14.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.761+0000 7fee71405640 1 -- start start 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.761+0000 7fee71405640 10 monclient: init 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 10 monclient(hunting): picked mon.noname-a con 0x7fee6c0a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 10 monclient(hunting): picked mon.noname-b con 0x7fee6c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 10 monclient(hunting): picked mon.noname-c con 0x7fee6c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c142520 con 0x7fee6c0a98f0 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c0b28a0 con 0x7fee6c1430c0 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c0b2a30 con 0x7fee6c13f710 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:14.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.762+0000 7fee71405640 10 monclient(hunting): authenticate will time out at 5728.025582s 2024-03-25T17:36:14.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.769+0000 7fee63fff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fee6c13f710 0x7fee6c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:47436/0 (socket says 172.21.15.134:47436) 2024-03-25T17:36:14.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.769+0000 7fee63fff640 1 -- 172.21.15.134:0/3939770309 learned_addr learned my addr 172.21.15.134:0/3939770309 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:14.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.788+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971437942 0 0) 0x7fee6c0b2a30 con 0x7fee6c13f710 2024-03-25T17:36:14.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.788+0000 7fee6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:14.788 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.788+0000 7fee6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:14.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.788+0000 7fee6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:14.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.788+0000 7fee6a7fc640 10 monclient(hunting): my global_id is 34502 2024-03-25T17:36:14.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.788+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee58003520 con 0x7fee6c13f710 2024-03-25T17:36:14.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.799+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566938052 0 0) 0x7fee6c142520 con 0x7fee6c0a98f0 2024-03-25T17:36:14.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.799+0000 7fee6a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:14.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.799+0000 7fee6a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:14.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.799+0000 7fee6a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:14.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.799+0000 7fee6a7fc640 10 monclient(hunting): my global_id is 4515 2024-03-25T17:36:14.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.799+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee6c0b2a30 con 0x7fee6c0a98f0 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.803+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3500390675 0 0) 0x7fee58003520 con 0x7fee6c13f710 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.803+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 >> v1:172.21.15.159:6789/0 conn(0x7fee6c1430c0 legacy=0x7fee6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.803+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 >> v1:172.21.15.134:6789/0 conn(0x7fee6c0a98f0 legacy=0x7fee6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.803+0000 7fee6a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.803+0000 7fee6a7fc640 1 monclient: found mon.noname-b 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c0b44c0 con 0x7fee6c13f710 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:14.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fee54001690 con 0x7fee6c13f710 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee6a7fc640 10 monclient: dump: 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:14.806 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:14.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee71405640 5 monclient: authenticate success, global_id 34502 2024-03-25T17:36:14.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.804+0000 7fee71405640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:14.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee540017f0 con 0x7fee6c13f710 2024-03-25T17:36:14.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee6a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:14.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee6a7fc640 1 -- 172.21.15.134:0/3939770309 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fee54001c00 con 0x7fee6c13f710 2024-03-25T17:36:14.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee71405640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:14.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee71405640 4 set_mon_vals no callback set 2024-03-25T17:36:14.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:14.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.813+0000 7fee6a7fc640 10 monclient: dump: 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:14.815 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:14.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.814+0000 7fee71405640 10 monclient: shutdown 2024-03-25T17:36:14.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.814+0000 7fee71405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:14.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.814+0000 7fee71405640 1 -- 172.21.15.134:0/3939770309 >> v1:172.21.15.134:6790/0 conn(0x7fee6c13f710 legacy=0x7fee6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:14.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.814+0000 7fee71405640 1 -- 172.21.15.134:0/3939770309 shutdown_connections 2024-03-25T17:36:14.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.814+0000 7fee71405640 1 -- 172.21.15.134:0/3939770309 >> 172.21.15.134:0/3939770309 conn(0x7fee6c0b7ea0 msgr2=0x7fee6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:14.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.815+0000 7fee71405640 1 -- 172.21.15.134:0/3939770309 shutdown_connections 2024-03-25T17:36:14.816 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.815+0000 7fee71405640 1 -- 172.21.15.134:0/3939770309 wait complete. 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.816+0000 7fee71405640 10 monclient: build_initial_monmap 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.816+0000 7fee71405640 10 monclient: monmap: 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:36:14.817 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:14.818 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:36:14.818 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:36:14.818 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:14.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.817+0000 7fee71405640 1 librados: starting msgr at 2024-03-25T17:36:14.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.817+0000 7fee71405640 1 librados: starting objecter 2024-03-25T17:36:14.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.817+0000 7fee71405640 1 Processor -- start 2024-03-25T17:36:14.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.817+0000 7fee71405640 1 -- start start 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.817+0000 7fee71405640 1 librados: setting wanted keys 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.817+0000 7fee71405640 1 librados: calling monclient init 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.817+0000 7fee71405640 10 monclient: init 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.818+0000 7fee71405640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.818+0000 7fee71405640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.818+0000 7fee71405640 10 monclient(hunting): picked mon.noname-c con 0x7fee6c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.818+0000 7fee71405640 10 monclient(hunting): picked mon.noname-a con 0x7fee6c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.819+0000 7fee71405640 10 monclient(hunting): picked mon.noname-b con 0x7fee6c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.819+0000 7fee71405640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c139a30 con 0x7fee6c13f710 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.819+0000 7fee71405640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c1dabc0 con 0x7fee6c1430c0 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.819+0000 7fee71405640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c1d41e0 con 0x7fee6c0a98f0 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.819+0000 7fee71405640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:14.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.819+0000 7fee71405640 10 monclient(hunting): authenticate will time out at 5728.081830s 2024-03-25T17:36:14.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.827+0000 7fee6b7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fee6c1430c0 0x7fee6c1d4740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:38590/0 (socket says 172.21.15.134:38590) 2024-03-25T17:36:14.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.827+0000 7fee6b7fe640 1 -- 172.21.15.134:0/1468770006 learned_addr learned my addr 172.21.15.134:0/1468770006 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:14.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.837+0000 7fee637fe640 10 client.?.objecter ms_handle_connect 0x7fee6c13f710 2024-03-25T17:36:14.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.837+0000 7fee637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:14.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.837+0000 7fee637fe640 10 client.?.objecter ms_handle_connect 0x7fee6c1430c0 2024-03-25T17:36:14.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.837+0000 7fee637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:14.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.838+0000 7fee637fe640 10 client.?.objecter ms_handle_connect 0x7fee6c0a98f0 2024-03-25T17:36:14.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.838+0000 7fee637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:14.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.841+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467401684 0 0) 0x7fee6c1dabc0 con 0x7fee6c1430c0 2024-03-25T17:36:14.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.841+0000 7fee637fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:14.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.841+0000 7fee637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:14.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.841+0000 7fee637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:14.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.841+0000 7fee637fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T17:36:14.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.841+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee4c004eb0 con 0x7fee6c1430c0 2024-03-25T17:36:14.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.850+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308815994 0 0) 0x7fee6c1d41e0 con 0x7fee6c0a98f0 2024-03-25T17:36:14.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.850+0000 7fee637fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:14.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.850+0000 7fee637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:14.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.850+0000 7fee637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:14.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.850+0000 7fee637fe640 10 monclient(hunting): my global_id is 34505 2024-03-25T17:36:14.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.850+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee6c1dabc0 con 0x7fee6c0a98f0 2024-03-25T17:36:14.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214793089 0 0) 0x7fee6c139a30 con 0x7fee6c13f710 2024-03-25T17:36:14.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:14.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:14.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:14.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): my global_id is 4518 2024-03-25T17:36:14.858 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee6c1d41e0 con 0x7fee6c13f710 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 602877334 0 0) 0x7fee4c004eb0 con 0x7fee6c1430c0 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee6c139a30 con 0x7fee6c1430c0 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fee64001c30 con 0x7fee6c1430c0 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): dump: 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:14.859 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:14.860 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:14.860 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:14.860 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:14.860 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.857+0000 7fee637fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.864+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4070169179 0 0) 0x7fee6c1dabc0 con 0x7fee6c0a98f0 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.864+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee4c004eb0 con 0x7fee6c0a98f0 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.864+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fee5c001930 con 0x7fee6c0a98f0 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.864+0000 7fee637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.864+0000 7fee637fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.864+0000 7fee637fe640 10 monclient(hunting): dump: 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:14.865 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:14.866 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:14.866 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1300273829 0 0) 0x7fee4c004eb0 con 0x7fee6c0a98f0 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 >> v1:172.21.15.159:6789/0 conn(0x7fee6c1430c0 legacy=0x7fee6c1d4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 >> v1:172.21.15.134:6789/0 conn(0x7fee6c13f710 legacy=0x7fee6c1db0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 10 monclient: _finish_hunting 0 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 1 monclient: found mon.c 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c1d4370 con 0x7fee6c0a98f0 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 10 monclient: _finish_auth 0 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 10 monclient: _check_auth_tickets 2024-03-25T17:36:14.872 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee71405640 5 monclient: authenticate success, global_id 34505 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee71405640 10 monclient: _renew_subs 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee71405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.870+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c139040 con 0x7fee6c0a98f0 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.871+0000 7fee71405640 10 client.34505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.871+0000 7fee71405640 10 monclient: _renew_subs 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.871+0000 7fee71405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.871+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c1db990 con 0x7fee6c0a98f0 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.871+0000 7fee71405640 1 librados: init done 2024-03-25T17:36:14.873 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.872+0000 7fee71405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:14.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.872+0000 7fee71405640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:14.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.872+0000 7fee71405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:14.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:14.872+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c0627b0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee5c0035f0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fee5c0037a0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee637fe640 10 monclient: dump: 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:15.173 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:15.174 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:15.174 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:15.174 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:15.174 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:15.174 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:15.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee69ffb640 4 set_mon_vals no callback set 2024-03-25T17:36:15.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.172+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fee5c0039c0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.173+0000 7fee63fff640 10 monclient: get_auth_request con 0x7fee4c06dfb0 auth_method 0 2024-03-25T17:36:15.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.173+0000 7fee637fe640 10 client.34505.objecter ms_handle_connect 0x7fee4c06dfb0 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fee5c0533b0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 10 client.34505.objecter ms_dispatch 0x7fee6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 3 client.34505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 3 client.34505.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 20 client.34505.objecter dump_active .. 0 homeless 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee5c04eec0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:15.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.184+0000 7fee637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:15.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.429+0000 7fee71405640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:15.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.429+0000 7fee71405640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:15.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.429+0000 7fee71405640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:15.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.429+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee6c13dfd0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.467+0000 7fee637fe640 1 -- 172.21.15.134:0/1468770006 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fee5c050df0 con 0x7fee6c0a98f0 2024-03-25T17:36:15.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.467+0000 7fee637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:15.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.467+0000 7fee637fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:15.469 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:15.469 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:15.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 10 librados: watch_flush enter 2024-03-25T17:36:15.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 10 librados: watch_flush exit 2024-03-25T17:36:15.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 20 client.34505.objecter shutdown clearing up homeless session... 2024-03-25T17:36:15.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 10 client.34505.objecter successfully canceled tick 2024-03-25T17:36:15.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 >> v1:172.21.15.134:6800/610837766 conn(0x7fee4c06dfb0 legacy=0x7fee4c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:15.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 10 monclient: shutdown 2024-03-25T17:36:15.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:15.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 >> v1:172.21.15.134:6790/0 conn(0x7fee6c0a98f0 legacy=0x7fee6c138930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:15.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 shutdown_connections 2024-03-25T17:36:15.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 >> 172.21.15.134:0/1468770006 conn(0x7fee6c0b7ea0 msgr2=0x7fee6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:15.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 shutdown_connections 2024-03-25T17:36:15.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 1 -- 172.21.15.134:0/1468770006 wait complete. 2024-03-25T17:36:15.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:15.470+0000 7fee71405640 1 librados: shutdown 2024-03-25T17:36:15.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:22.291 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:22.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.405+0000 7fc28ab15640 10 monclient: get_monmap_and_config 2024-03-25T17:36:22.405 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.405+0000 7fc28ab15640 10 monclient: build_initial_monmap 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.405+0000 7fc28ab15640 10 monclient: monmap: 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:36:22.406063+0000 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:36:22.406063+0000 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:36:22.406 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.405+0000 7fc28ab15640 1 Processor -- start 2024-03-25T17:36:22.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 1 -- start start 2024-03-25T17:36:22.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient: init 2024-03-25T17:36:22.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient(hunting): picked mon.noname-b con 0x7fc2840b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient(hunting): picked mon.noname-a con 0x7fc2840b1730 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient(hunting): picked mon.noname-c con 0x7fc28414be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2840af940 con 0x7fc2840b1730 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2840b0a70 con 0x7fc28414be90 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2840b0d50 con 0x7fc2840b4d60 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:22.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.406+0000 7fc28ab15640 10 monclient(hunting): authenticate will time out at 5735.669670s 2024-03-25T17:36:22.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.413+0000 7fc2890b2640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc28414be90 0x7fc28414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:50342/0 (socket says 172.21.15.134:50342) 2024-03-25T17:36:22.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.413+0000 7fc2890b2640 1 -- 172.21.15.134:0/3999192593 learned_addr learned my addr 172.21.15.134:0/3999192593 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:22.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.428+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021087003 0 0) 0x7fc2840b0a70 con 0x7fc28414be90 2024-03-25T17:36:22.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.428+0000 7fc2837fe640 10 monclient(hunting): _negotiate 2024-03-25T17:36:22.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.428+0000 7fc2837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:22.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.428+0000 7fc2837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:22.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.428+0000 7fc2837fe640 10 monclient(hunting): my global_id is 4543 2024-03-25T17:36:22.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.428+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc274003520 con 0x7fc28414be90 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2301313783 0 0) 0x7fc274003520 con 0x7fc28414be90 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 >> v1:172.21.15.134:6790/0 conn(0x7fc2840b4d60 legacy=0x7fc28413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 >> v1:172.21.15.134:6789/0 conn(0x7fc2840b1730 legacy=0x7fc2840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 10 monclient: _finish_hunting 0 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 1 monclient: found mon.noname-c 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2840b1030 con 0x7fc28414be90 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 10 monclient: _finish_auth 0 2024-03-25T17:36:22.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 10 monclient: _check_auth_tickets 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc27c001a10 con 0x7fc28414be90 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc2837fe640 10 monclient: dump: 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:22.441 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc28ab15640 5 monclient: authenticate success, global_id 4543 2024-03-25T17:36:22.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.439+0000 7fc28ab15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc27c001be0 con 0x7fc28414be90 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc2837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc2837fe640 1 -- 172.21.15.134:0/3999192593 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc27c002030 con 0x7fc28414be90 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc28ab15640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc28ab15640 4 set_mon_vals no callback set 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc2837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc2837fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.453+0000 7fc2837fe640 10 monclient: dump: 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:22.454 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:22.455 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.454+0000 7fc28ab15640 10 monclient: shutdown 2024-03-25T17:36:22.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.454+0000 7fc28ab15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:22.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.454+0000 7fc28ab15640 1 -- 172.21.15.134:0/3999192593 >> v1:172.21.15.159:6789/0 conn(0x7fc28414be90 legacy=0x7fc28414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.454+0000 7fc28ab15640 1 -- 172.21.15.134:0/3999192593 shutdown_connections 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.454+0000 7fc28ab15640 1 -- 172.21.15.134:0/3999192593 >> 172.21.15.134:0/3999192593 conn(0x7fc2840617a0 msgr2=0x7fc284061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.454+0000 7fc28ab15640 1 -- 172.21.15.134:0/3999192593 shutdown_connections 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.454+0000 7fc28ab15640 1 -- 172.21.15.134:0/3999192593 wait complete. 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.455+0000 7fc28ab15640 10 monclient: build_initial_monmap 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.455+0000 7fc28ab15640 10 monclient: monmap: 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:22.457 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 1 librados: starting msgr at 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 1 librados: starting objecter 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 1 Processor -- start 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 1 -- start start 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 1 librados: setting wanted keys 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 1 librados: calling monclient init 2024-03-25T17:36:22.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 10 monclient: init 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.456+0000 7fc28ab15640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 10 monclient(hunting): picked mon.noname-b con 0x7fc2840b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 10 monclient(hunting): picked mon.noname-c con 0x7fc2840b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 10 monclient(hunting): picked mon.noname-a con 0x7fc28414be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28413a910 con 0x7fc28414be90 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc284139430 con 0x7fc2840b1730 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc284138680 con 0x7fc2840b4d60 2024-03-25T17:36:22.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:22.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.457+0000 7fc28ab15640 10 monclient(hunting): authenticate will time out at 5735.719838s 2024-03-25T17:36:22.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.469+0000 7fc2888b1640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc2840b1730 0x7fc284131f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:50370/0 (socket says 172.21.15.134:50370) 2024-03-25T17:36:22.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.469+0000 7fc2888b1640 1 -- 172.21.15.134:0/1045694438 learned_addr learned my addr 172.21.15.134:0/1045694438 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:22.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.471+0000 7fc2817fa640 10 client.?.objecter ms_handle_connect 0x7fc28414be90 2024-03-25T17:36:22.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.471+0000 7fc2817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:22.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.472+0000 7fc2817fa640 10 client.?.objecter ms_handle_connect 0x7fc2840b1730 2024-03-25T17:36:22.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.472+0000 7fc2817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:22.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.482+0000 7fc2817fa640 10 client.?.objecter ms_handle_connect 0x7fc2840b4d60 2024-03-25T17:36:22.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.482+0000 7fc2817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:22.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.483+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107303661 0 0) 0x7fc28413a910 con 0x7fc28414be90 2024-03-25T17:36:22.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.483+0000 7fc2817fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:22.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.483+0000 7fc2817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:22.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.483+0000 7fc2817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:22.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.483+0000 7fc2817fa640 10 monclient(hunting): my global_id is 4521 2024-03-25T17:36:22.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.483+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc268004eb0 con 0x7fc28414be90 2024-03-25T17:36:22.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.489+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209075465 0 0) 0x7fc284139430 con 0x7fc2840b1730 2024-03-25T17:36:22.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.489+0000 7fc2817fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:22.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.489+0000 7fc2817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:22.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.489+0000 7fc2817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:22.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.489+0000 7fc2817fa640 10 monclient(hunting): my global_id is 4546 2024-03-25T17:36:22.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.489+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28413a910 con 0x7fc2840b1730 2024-03-25T17:36:22.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.493+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893780149 0 0) 0x7fc284138680 con 0x7fc2840b4d60 2024-03-25T17:36:22.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.493+0000 7fc2817fa640 10 monclient(hunting): _negotiate 2024-03-25T17:36:22.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.493+0000 7fc2817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:22.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.493+0000 7fc2817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:22.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.493+0000 7fc2817fa640 10 monclient(hunting): my global_id is 34511 2024-03-25T17:36:22.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.493+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc284139430 con 0x7fc2840b4d60 2024-03-25T17:36:22.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.496+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2217279722 0 0) 0x7fc268004eb0 con 0x7fc28414be90 2024-03-25T17:36:22.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.496+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc284138680 con 0x7fc28414be90 2024-03-25T17:36:22.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.496+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc27c001ef0 con 0x7fc28414be90 2024-03-25T17:36:22.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.496+0000 7fc2817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:22.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.496+0000 7fc2817fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:22.497 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.496+0000 7fc2817fa640 10 monclient(hunting): dump: 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.498 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.496+0000 7fc2817fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:22.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.513+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745440446 0 0) 0x7fc284139430 con 0x7fc2840b4d60 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.514+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc268004eb0 con 0x7fc2840b4d60 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.514+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc270001510 con 0x7fc2840b4d60 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.514+0000 7fc2817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.514+0000 7fc2817fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.514+0000 7fc2817fa640 10 monclient(hunting): dump: 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:22.515 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:22.516 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:22.516 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.515+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3567668111 0 0) 0x7fc28413a910 con 0x7fc2840b1730 2024-03-25T17:36:22.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.515+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc284139430 con 0x7fc2840b1730 2024-03-25T17:36:22.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.515+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc2780019b0 con 0x7fc2840b1730 2024-03-25T17:36:22.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.515+0000 7fc2817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.515+0000 7fc2817fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.515+0000 7fc2817fa640 10 monclient(hunting): dump: 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:22.517 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1201962349 0 0) 0x7fc284138680 con 0x7fc28414be90 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 >> v1:172.21.15.134:6790/0 conn(0x7fc2840b4d60 legacy=0x7fc2841dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 >> v1:172.21.15.159:6789/0 conn(0x7fc2840b1730 legacy=0x7fc284131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 10 monclient: _finish_hunting 0 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 1 monclient: found mon.a 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc284138810 con 0x7fc28414be90 2024-03-25T17:36:22.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 10 monclient: _finish_auth 0 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 10 monclient: _check_auth_tickets 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc2817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 5 monclient: authenticate success, global_id 4521 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 10 monclient: _renew_subs 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc284132620 con 0x7fc28414be90 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 10 monclient: _renew_subs 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2841453f0 con 0x7fc28414be90 2024-03-25T17:36:22.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.521+0000 7fc28ab15640 1 librados: init done 2024-03-25T17:36:22.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.522+0000 7fc28ab15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:22.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.522+0000 7fc28ab15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:22.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.522+0000 7fc28ab15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:36:22.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.522+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2841401f0 con 0x7fc28414be90 2024-03-25T17:36:22.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.820+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc27c002090 con 0x7fc28414be90 2024-03-25T17:36:22.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.820+0000 7fc2817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.820+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc27c0015d0 con 0x7fc28414be90 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.820+0000 7fc2817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.820+0000 7fc2817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.820+0000 7fc2817fa640 10 monclient: dump: 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:22.821 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:22.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.820+0000 7fc282ffd640 4 set_mon_vals no callback set 2024-03-25T17:36:22.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.821+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fc27c01ae20 con 0x7fc28414be90 2024-03-25T17:36:22.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.821+0000 7fc2888b1640 10 monclient: get_auth_request con 0x7fc26806e070 auth_method 0 2024-03-25T17:36:22.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.822+0000 7fc2817fa640 10 client.4521.objecter ms_handle_connect 0x7fc26806e070 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fc27c054a60 con 0x7fc28414be90 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 10 client.4521.objecter ms_dispatch 0x7fc284000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc27c0503a0 con 0x7fc28414be90 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:22.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:22.845+0000 7fc2817fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:23.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.075+0000 7fc28ab15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:23.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.075+0000 7fc28ab15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:23.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.075+0000 7fc28ab15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:36:23.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.075+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2840afbd0 con 0x7fc28414be90 2024-03-25T17:36:23.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.102+0000 7fc2817fa640 1 -- 172.21.15.134:0/1045694438 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fc27c0524a0 con 0x7fc28414be90 2024-03-25T17:36:23.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.102+0000 7fc2817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:23.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.102+0000 7fc2817fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:23.103 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:23.103 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:23.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 10 librados: watch_flush enter 2024-03-25T17:36:23.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 10 librados: watch_flush exit 2024-03-25T17:36:23.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T17:36:23.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 10 client.4521.objecter successfully canceled tick 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 >> v1:172.21.15.134:6800/610837766 conn(0x7fc26806e070 legacy=0x7fc268070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 10 monclient: shutdown 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 >> v1:172.21.15.134:6789/0 conn(0x7fc28414be90 legacy=0x7fc2841db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 shutdown_connections 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 >> 172.21.15.134:0/1045694438 conn(0x7fc2840617a0 msgr2=0x7fc284061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 shutdown_connections 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 1 -- 172.21.15.134:0/1045694438 wait complete. 2024-03-25T17:36:23.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:23.105+0000 7fc28ab15640 1 librados: shutdown 2024-03-25T17:36:23.119 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:29.927 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:30.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.043+0000 7f1f932b3640 10 monclient: get_monmap_and_config 2024-03-25T17:36:30.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.043+0000 7f1f932b3640 10 monclient: build_initial_monmap 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.043+0000 7f1f932b3640 10 monclient: monmap: 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:36:30.044889+0000 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:36:30.044889+0000 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:36:30.045 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:30.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.044+0000 7f1f932b3640 1 Processor -- start 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.044+0000 7f1f932b3640 1 -- start start 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.044+0000 7f1f932b3640 10 monclient: init 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c142520 con 0x7f1f8c13f710 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0b28a0 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c0b2a30 con 0x7f1f8c1430c0 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:30.047 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.045+0000 7f1f932b3640 10 monclient(hunting): authenticate will time out at 5743.308235s 2024-03-25T17:36:30.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.052+0000 7f1f9084e640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f1f8c13f710 0x7f1f8c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34604/0 (socket says 172.21.15.134:34604) 2024-03-25T17:36:30.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.052+0000 7f1f9084e640 1 -- 172.21.15.134:0/2994129261 learned_addr learned my addr 172.21.15.134:0/2994129261 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:30.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.071+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162416348 0 0) 0x7f1f8c0b2a30 con 0x7f1f8c1430c0 2024-03-25T17:36:30.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.071+0000 7f1f83fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:30.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.071+0000 7f1f83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:30.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.071+0000 7f1f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:30.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.071+0000 7f1f83fff640 10 monclient(hunting): my global_id is 34514 2024-03-25T17:36:30.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.071+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f78003520 con 0x7f1f8c1430c0 2024-03-25T17:36:30.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.073+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2419828379 0 0) 0x7f1f8c142520 con 0x7f1f8c13f710 2024-03-25T17:36:30.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.073+0000 7f1f83fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:30.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.073+0000 7f1f83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:30.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.073+0000 7f1f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:30.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.073+0000 7f1f83fff640 10 monclient(hunting): my global_id is 4524 2024-03-25T17:36:30.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.073+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f8c0b2a30 con 0x7f1f8c13f710 2024-03-25T17:36:30.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.086+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186053946 0 0) 0x7f1f8c0b28a0 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.086+0000 7f1f83fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:30.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.086+0000 7f1f83fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:30.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.086+0000 7f1f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:30.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.086+0000 7f1f83fff640 10 monclient(hunting): my global_id is 4549 2024-03-25T17:36:30.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.086+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f8c142520 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1676147291 0 0) 0x7f1f78003520 con 0x7f1f8c1430c0 2024-03-25T17:36:30.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 >> v1:172.21.15.159:6789/0 conn(0x7f1f8c0a98f0 legacy=0x7f1f8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:30.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 >> v1:172.21.15.134:6789/0 conn(0x7f1f8c13f710 legacy=0x7f1f8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:30.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 10 monclient: _finish_hunting 0 2024-03-25T17:36:30.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 1 monclient: found mon.noname-b 2024-03-25T17:36:30.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:30.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c0b44c0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 10 monclient: _finish_auth 0 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 10 monclient: _check_auth_tickets 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1f84001a10 con 0x7f1f8c1430c0 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f83fff640 10 monclient: dump: 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.090 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:30.091 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:30.091 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:30.091 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:30.091 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:30.091 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:30.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f932b3640 5 monclient: authenticate success, global_id 34514 2024-03-25T17:36:30.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.088+0000 7f1f932b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:30.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f84001b70 con 0x7f1f8c1430c0 2024-03-25T17:36:30.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f83fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:30.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f83fff640 1 -- 172.21.15.134:0/2994129261 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1f84001f80 con 0x7f1f8c1430c0 2024-03-25T17:36:30.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:30.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:30.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f83fff640 10 monclient: dump: 2024-03-25T17:36:30.103 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f932b3640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:30.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.102+0000 7f1f932b3640 4 set_mon_vals no callback set 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.103+0000 7f1f932b3640 10 monclient: shutdown 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.103+0000 7f1f932b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.103+0000 7f1f932b3640 1 -- 172.21.15.134:0/2994129261 >> v1:172.21.15.134:6790/0 conn(0x7f1f8c1430c0 legacy=0x7f1f8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.103+0000 7f1f932b3640 1 -- 172.21.15.134:0/2994129261 shutdown_connections 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.103+0000 7f1f932b3640 1 -- 172.21.15.134:0/2994129261 >> 172.21.15.134:0/2994129261 conn(0x7f1f8c0b7ea0 msgr2=0x7f1f8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.104+0000 7f1f932b3640 1 -- 172.21.15.134:0/2994129261 shutdown_connections 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.104+0000 7f1f932b3640 1 -- 172.21.15.134:0/2994129261 wait complete. 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.104+0000 7f1f932b3640 10 monclient: build_initial_monmap 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.104+0000 7f1f932b3640 10 monclient: monmap: 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:36:30.106 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.105+0000 7f1f932b3640 1 librados: starting msgr at 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.105+0000 7f1f932b3640 1 librados: starting objecter 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.105+0000 7f1f932b3640 1 Processor -- start 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.105+0000 7f1f932b3640 1 -- start start 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.105+0000 7f1f932b3640 1 librados: setting wanted keys 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.105+0000 7f1f932b3640 1 librados: calling monclient init 2024-03-25T17:36:30.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.105+0000 7f1f932b3640 10 monclient: init 2024-03-25T17:36:30.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:30.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 10 monclient(hunting): picked mon.noname-b con 0x7f1f8c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 10 monclient(hunting): picked mon.noname-c con 0x7f1f8c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c137ab0 con 0x7f1f8c13f710 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c138bd0 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8c138e40 con 0x7f1f8c1430c0 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:30.109 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.106+0000 7f1f932b3640 10 monclient(hunting): authenticate will time out at 5743.369032s 2024-03-25T17:36:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.109+0000 7f1f91850640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f1f8c1430c0 0x7f1f8c1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:38464/0 (socket says 172.21.15.134:38464) 2024-03-25T17:36:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.109+0000 7f1f91850640 1 -- 172.21.15.134:0/1995142128 learned_addr learned my addr 172.21.15.134:0/1995142128 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.110+0000 7f1f81ffb640 10 client.?.objecter ms_handle_connect 0x7f1f8c1430c0 2024-03-25T17:36:30.110 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.110+0000 7f1f81ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:30.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.132+0000 7f1f81ffb640 10 client.?.objecter ms_handle_connect 0x7f1f8c0a98f0 2024-03-25T17:36:30.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.132+0000 7f1f81ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:30.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.132+0000 7f1f81ffb640 10 client.?.objecter ms_handle_connect 0x7f1f8c13f710 2024-03-25T17:36:30.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.132+0000 7f1f81ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:30.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.136+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064057158 0 0) 0x7f1f8c138e40 con 0x7f1f8c1430c0 2024-03-25T17:36:30.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.136+0000 7f1f81ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:30.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.136+0000 7f1f81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:30.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.136+0000 7f1f81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:30.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.136+0000 7f1f81ffb640 10 monclient(hunting): my global_id is 34517 2024-03-25T17:36:30.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.136+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f70004eb0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.142 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.142+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843081730 0 0) 0x7f1f8c138bd0 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.142+0000 7f1f81ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:30.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.142+0000 7f1f81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:30.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.142+0000 7f1f81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:30.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.142+0000 7f1f81ffb640 10 monclient(hunting): my global_id is 4552 2024-03-25T17:36:30.143 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.142+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f8c138e40 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.148+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745274609 0 0) 0x7f1f70004eb0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.148+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f8c138bd0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.148+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1f84001ec0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.148+0000 7f1f81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.148+0000 7f1f81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.148+0000 7f1f81ffb640 10 monclient(hunting): dump: 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:30.149 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:30.150 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.148+0000 7f1f81ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:30.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.151+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830391208 0 0) 0x7f1f8c137ab0 con 0x7f1f8c13f710 2024-03-25T17:36:30.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.151+0000 7f1f81ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:30.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.151+0000 7f1f81ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:30.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.151+0000 7f1f81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:30.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.151+0000 7f1f81ffb640 10 monclient(hunting): my global_id is 4527 2024-03-25T17:36:30.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.151+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f70004eb0 con 0x7f1f8c13f710 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.162+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3500115940 0 0) 0x7f1f8c138e40 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.162+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f8c137ab0 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.162+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1f88001870 con 0x7f1f8c0a98f0 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.162+0000 7f1f81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.162+0000 7f1f81ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.162+0000 7f1f81ffb640 10 monclient(hunting): dump: 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:30.163 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.164 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.164 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:30.164 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:30.164 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:30.164 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:30.164 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:30.164 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3754883916 0 0) 0x7f1f8c138bd0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 >> v1:172.21.15.159:6789/0 conn(0x7f1f8c0a98f0 legacy=0x7f1f8c137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 >> v1:172.21.15.134:6789/0 conn(0x7f1f8c13f710 legacy=0x7f1f8c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 1 monclient: found mon.c 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:30.168 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f8c139120 con 0x7f1f8c1430c0 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 10 monclient: _finish_auth 0 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 5 monclient: authenticate success, global_id 34517 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 10 monclient: _renew_subs 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f8c139400 con 0x7f1f8c1430c0 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 10 client.34517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 10 monclient: _renew_subs 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f8c0b7880 con 0x7f1f8c1430c0 2024-03-25T17:36:30.169 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.167+0000 7f1f932b3640 1 librados: init done 2024-03-25T17:36:30.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.168+0000 7f1f932b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:30.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.168+0000 7f1f932b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:30.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.169+0000 7f1f932b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:30.170 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.169+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8c137810 con 0x7f1f8c1430c0 2024-03-25T17:36:30.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.475+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f84003c20 con 0x7f1f8c1430c0 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.475+0000 7f1f81ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.475+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f1f84002100 con 0x7f1f8c1430c0 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.475+0000 7f1f81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.475+0000 7f1f81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.475+0000 7f1f81ffb640 10 monclient: dump: 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:30.476 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:30.477 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:30.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.475+0000 7f1f837fe640 4 set_mon_vals no callback set 2024-03-25T17:36:30.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.476+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f1f840040f0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.476+0000 7f1f9104f640 10 monclient: get_auth_request con 0x7f1f7006dfb0 auth_method 0 2024-03-25T17:36:30.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.477+0000 7f1f81ffb640 10 client.34517.objecter ms_handle_connect 0x7f1f7006dfb0 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f1f84054af0 con 0x7f1f8c1430c0 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 10 client.34517.objecter ms_dispatch 0x7f1f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 3 client.34517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 3 client.34517.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 20 client.34517.objecter dump_active .. 0 homeless 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f84050600 con 0x7f1f8c1430c0 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:30.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.482+0000 7f1f81ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:30.590 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:35.606 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.621 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:35.637 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.636 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:35.653 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.647 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:35.676 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.649 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:35.665 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.653 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:35.681 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.658 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:35.675 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.677 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:35.706 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.703 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:35.720 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:30.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.705+0000 7f1f932b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:30.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.705+0000 7f1f932b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:30.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.705+0000 7f1f932b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:30.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.705+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f8c137c60 con 0x7f1f8c1430c0 2024-03-25T17:36:30.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.914+0000 7f1f81ffb640 1 -- 172.21.15.134:0/1995142128 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f1f8c137c60 con 0x7f1f8c1430c0 2024-03-25T17:36:30.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.914+0000 7f1f81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:30.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.914+0000 7f1f81ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:30.915 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:30.915 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 10 librados: watch_flush enter 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 10 librados: watch_flush exit 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 20 client.34517.objecter shutdown clearing up homeless session... 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 10 client.34517.objecter successfully canceled tick 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 >> v1:172.21.15.134:6800/610837766 conn(0x7f1f7006dfb0 legacy=0x7f1f70070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 10 monclient: shutdown 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 >> v1:172.21.15.134:6790/0 conn(0x7f1f8c1430c0 legacy=0x7f1f8c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 shutdown_connections 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 >> 172.21.15.134:0/1995142128 conn(0x7f1f8c0b7ea0 msgr2=0x7f1f8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 shutdown_connections 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 1 -- 172.21.15.134:0/1995142128 wait complete. 2024-03-25T17:36:30.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:30.917+0000 7f1f932b3640 1 librados: shutdown 2024-03-25T17:36:30.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:37.735 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.847+0000 7f2e139fb640 10 monclient: get_monmap_and_config 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.847+0000 7f2e139fb640 10 monclient: build_initial_monmap 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.847+0000 7f2e139fb640 10 monclient: monmap: 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:36:37.849721+0000 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:36:37.849721+0000 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:37.849 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:37.850 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:37.850 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:36:37.850 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:36:37.850 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:37.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.848+0000 7f2e139fb640 1 Processor -- start 2024-03-25T17:36:37.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.848+0000 7f2e139fb640 1 -- start start 2024-03-25T17:36:37.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.848+0000 7f2e139fb640 10 monclient: init 2024-03-25T17:36:37.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:37.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2e0c061830 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0c13c9f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 10 monclient(hunting): picked mon.noname-b con 0x7f2e0c072af0 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c071fc0 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c0707a0 con 0x7f2e0c061830 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c0ae9e0 con 0x7f2e0c072af0 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:37.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.849+0000 7f2e139fb640 10 monclient(hunting): authenticate will time out at 5751.113518s 2024-03-25T17:36:37.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.867+0000 7f2e11f98640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f2e0c072af0 0x7f2e0c14a890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:43258/0 (socket says 172.21.15.134:43258) 2024-03-25T17:36:37.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.867+0000 7f2e11f98640 1 -- 172.21.15.134:0/2704162778 learned_addr learned my addr 172.21.15.134:0/2704162778 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:37.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.880+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322747246 0 0) 0x7f2e0c071fc0 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.880+0000 7f2e03fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:37.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.880+0000 7f2e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:37.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.880+0000 7f2e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:37.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.880+0000 7f2e03fff640 10 monclient(hunting): my global_id is 4530 2024-03-25T17:36:37.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.880+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df8003520 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.881+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283784162 0 0) 0x7f2e0c0ae9e0 con 0x7f2e0c072af0 2024-03-25T17:36:37.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.881+0000 7f2e03fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:37.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.881+0000 7f2e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:37.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.881+0000 7f2e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:37.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.881+0000 7f2e03fff640 10 monclient(hunting): my global_id is 34520 2024-03-25T17:36:37.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.881+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e0c071fc0 con 0x7f2e0c072af0 2024-03-25T17:36:37.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.895+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910340560 0 0) 0x7f2e0c0707a0 con 0x7f2e0c061830 2024-03-25T17:36:37.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.895+0000 7f2e03fff640 10 monclient(hunting): _negotiate 2024-03-25T17:36:37.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.895+0000 7f2e03fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:37.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.895+0000 7f2e03fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:37.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.895+0000 7f2e03fff640 10 monclient(hunting): my global_id is 4555 2024-03-25T17:36:37.897 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.895+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e0c0ae9e0 con 0x7f2e0c061830 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1371166864 0 0) 0x7f2df8003520 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 >> v1:172.21.15.134:6790/0 conn(0x7f2e0c072af0 legacy=0x7f2e0c14a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 >> v1:172.21.15.159:6789/0 conn(0x7f2e0c061830 legacy=0x7f2e0c13c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 10 monclient: _finish_hunting 0 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 1 monclient: found mon.noname-a 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c070cc0 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 10 monclient: _finish_auth 0 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 10 monclient: _check_auth_tickets 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:37.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2dfc001690 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e03fff640 10 monclient: dump: 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e139fb640 5 monclient: authenticate success, global_id 4530 2024-03-25T17:36:37.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.901+0000 7f2e139fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dfc0017f0 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e03fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e03fff640 1 -- 172.21.15.134:0/2704162778 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2dfc001c00 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e03fff640 10 monclient: dump: 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.916 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:37.917 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:37.917 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:37.917 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:37.917 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:37.917 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:37.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e139fb640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:37.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.914+0000 7f2e139fb640 4 set_mon_vals no callback set 2024-03-25T17:36:37.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.915+0000 7f2e139fb640 10 monclient: shutdown 2024-03-25T17:36:37.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.915+0000 7f2e139fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.915+0000 7f2e139fb640 1 -- 172.21.15.134:0/2704162778 >> v1:172.21.15.134:6789/0 conn(0x7f2e0c13c9f0 legacy=0x7f2e0c0723e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.915+0000 7f2e139fb640 1 -- 172.21.15.134:0/2704162778 shutdown_connections 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.915+0000 7f2e139fb640 1 -- 172.21.15.134:0/2704162778 >> 172.21.15.134:0/2704162778 conn(0x7f2e0c0b0ef0 msgr2=0x7f2e0c0b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.915+0000 7f2e139fb640 1 -- 172.21.15.134:0/2704162778 shutdown_connections 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.915+0000 7f2e139fb640 1 -- 172.21.15.134:0/2704162778 wait complete. 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.916+0000 7f2e139fb640 10 monclient: build_initial_monmap 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.916+0000 7f2e139fb640 10 monclient: monmap: 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:37.919 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.916+0000 7f2e139fb640 1 librados: starting msgr at 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.916+0000 7f2e139fb640 1 librados: starting objecter 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 1 Processor -- start 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 1 -- start start 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 1 librados: setting wanted keys 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 1 librados: calling monclient init 2024-03-25T17:36:37.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 10 monclient: init 2024-03-25T17:36:37.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:37.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:36:37.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.917+0000 7f2e139fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2e0c061830 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:37.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.918+0000 7f2e139fb640 10 monclient(hunting): picked mon.noname-b con 0x7f2e0c072af0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:37.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.918+0000 7f2e139fb640 10 monclient(hunting): picked mon.noname-a con 0x7f2e0c13c9f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:37.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.918+0000 7f2e139fb640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c141f90 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.918+0000 7f2e139fb640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c142200 con 0x7f2e0c072af0 2024-03-25T17:36:37.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.918+0000 7f2e139fb640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c1424e0 con 0x7f2e0c061830 2024-03-25T17:36:37.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.918+0000 7f2e139fb640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:37.922 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.918+0000 7f2e139fb640 10 monclient(hunting): authenticate will time out at 5751.181967s 2024-03-25T17:36:37.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.926+0000 7f2e11f98640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f2e0c13c9f0 0x7f2e0c1d92a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:39538/0 (socket says 172.21.15.134:39538) 2024-03-25T17:36:37.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.926+0000 7f2e11f98640 1 -- 172.21.15.134:0/3971274627 learned_addr learned my addr 172.21.15.134:0/3971274627 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:37.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.928+0000 7f2e01ffb640 10 client.?.objecter ms_handle_connect 0x7f2e0c13c9f0 2024-03-25T17:36:37.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.928+0000 7f2e01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:37.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.929+0000 7f2e01ffb640 10 client.?.objecter ms_handle_connect 0x7f2e0c061830 2024-03-25T17:36:37.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.929+0000 7f2e01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:37.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.930+0000 7f2e01ffb640 10 client.?.objecter ms_handle_connect 0x7f2e0c072af0 2024-03-25T17:36:37.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.930+0000 7f2e01ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:37.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.940+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238188006 0 0) 0x7f2e0c141f90 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.940+0000 7f2e01ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:37.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.940+0000 7f2e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:37.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.940+0000 7f2e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:37.942 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.940+0000 7f2e01ffb640 10 monclient(hunting): my global_id is 4533 2024-03-25T17:36:37.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.940+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0004eb0 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.944+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060647305 0 0) 0x7f2e0c1424e0 con 0x7f2e0c061830 2024-03-25T17:36:37.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.944+0000 7f2e01ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:37.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.944+0000 7f2e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:37.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.944+0000 7f2e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:37.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.944+0000 7f2e01ffb640 10 monclient(hunting): my global_id is 34523 2024-03-25T17:36:37.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.944+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e0c141f90 con 0x7f2e0c061830 2024-03-25T17:36:37.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.945+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395020844 0 0) 0x7f2e0c142200 con 0x7f2e0c072af0 2024-03-25T17:36:37.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.945+0000 7f2e01ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:36:37.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.945+0000 7f2e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:37.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.945+0000 7f2e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:37.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.945+0000 7f2e01ffb640 10 monclient(hunting): my global_id is 4558 2024-03-25T17:36:37.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.945+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e0c1424e0 con 0x7f2e0c072af0 2024-03-25T17:36:37.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.960+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217986837 0 0) 0x7f2e0c141f90 con 0x7f2e0c061830 2024-03-25T17:36:37.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.960+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e0c142200 con 0x7f2e0c061830 2024-03-25T17:36:37.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.960+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2e08001870 con 0x7f2e0c061830 2024-03-25T17:36:37.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.960+0000 7f2e01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.960+0000 7f2e01ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.960+0000 7f2e01ffb640 10 monclient(hunting): dump: 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:37.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.960+0000 7f2e01ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:37.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.963+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2929967288 0 0) 0x7f2df0004eb0 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.963+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e0c141f90 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.963+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2e04001cf0 con 0x7f2e0c13c9f0 2024-03-25T17:36:37.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.963+0000 7f2e01ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:37.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.963+0000 7f2e01ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:36:37.965 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.963+0000 7f2e01ffb640 10 monclient(hunting): dump: 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:37.966 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:37.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.967+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3025183147 0 0) 0x7f2e0c142200 con 0x7f2e0c061830 2024-03-25T17:36:37.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 >> v1:172.21.15.159:6789/0 conn(0x7f2e0c072af0 legacy=0x7f2e0c1d8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:37.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 >> v1:172.21.15.134:6789/0 conn(0x7f2e0c13c9f0 legacy=0x7f2e0c1d92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:37.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:36:37.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 1 monclient: found mon.c 2024-03-25T17:36:37.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:37.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:37.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c0b3e20 con 0x7f2e0c061830 2024-03-25T17:36:37.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 10 monclient: _finish_auth 0 2024-03-25T17:36:37.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:36:37.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:37.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 5 monclient: authenticate success, global_id 34523 2024-03-25T17:36:37.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 10 monclient: _renew_subs 2024-03-25T17:36:37.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:37.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0c142f50 con 0x7f2e0c061830 2024-03-25T17:36:37.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 10 client.34523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:37.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 10 monclient: _renew_subs 2024-03-25T17:36:37.972 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:37.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c0b3800 con 0x7f2e0c061830 2024-03-25T17:36:37.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.968+0000 7f2e139fb640 1 librados: init done 2024-03-25T17:36:37.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.969+0000 7f2e139fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:37.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.969+0000 7f2e139fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:37.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.969+0000 7f2e139fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:37.973 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:37.969+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c0627b0 con 0x7f2e0c061830 2024-03-25T17:36:38.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.196+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e08003610 con 0x7f2e0c061830 2024-03-25T17:36:38.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.196+0000 7f2e01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:38.198 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.196+0000 7f2e037fe640 4 set_mon_vals no callback set 2024-03-25T17:36:38.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.279+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f2e08001d80 con 0x7f2e0c061830 2024-03-25T17:36:38.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.279+0000 7f2e01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.279+0000 7f2e01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.279+0000 7f2e01ffb640 10 monclient: dump: 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:38.281 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:38.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.279+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f2e08003ae0 con 0x7f2e0c061830 2024-03-25T17:36:38.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.280+0000 7f2e10f96640 10 monclient: get_auth_request con 0x7f2df006df70 auth_method 0 2024-03-25T17:36:38.282 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.280+0000 7f2e01ffb640 10 client.34523.objecter ms_handle_connect 0x7f2df006df70 2024-03-25T17:36:38.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f2e080535b0 con 0x7f2e0c061830 2024-03-25T17:36:38.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 10 client.34523.objecter ms_dispatch 0x7f2e0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:38.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 3 client.34523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:38.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 3 client.34523.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:38.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 20 client.34523.objecter dump_active .. 0 homeless 2024-03-25T17:36:38.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e0804f0c0 con 0x7f2e0c061830 2024-03-25T17:36:38.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:38.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.287+0000 7f2e01ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:38.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.503+0000 7f2e139fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:38.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.503+0000 7f2e139fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:38.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.503+0000 7f2e139fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:36:38.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.503+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e0c1360a0 con 0x7f2e0c061830 2024-03-25T17:36:38.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.622+0000 7f2e01ffb640 1 -- 172.21.15.134:0/3971274627 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f2e08050ff0 con 0x7f2e0c061830 2024-03-25T17:36:38.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.622+0000 7f2e01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:38.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.622+0000 7f2e01ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:38.624 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:38.624 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.624+0000 7f2e139fb640 10 librados: watch_flush enter 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.624+0000 7f2e139fb640 10 librados: watch_flush exit 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.624+0000 7f2e139fb640 20 client.34523.objecter shutdown clearing up homeless session... 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.624+0000 7f2e139fb640 10 client.34523.objecter successfully canceled tick 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 >> v1:172.21.15.134:6800/610837766 conn(0x7f2df006df70 legacy=0x7f2df0070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 10 monclient: shutdown 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 >> v1:172.21.15.134:6790/0 conn(0x7f2e0c061830 legacy=0x7f2e0c12e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 shutdown_connections 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 >> 172.21.15.134:0/3971274627 conn(0x7f2e0c0b0ef0 msgr2=0x7f2e0c0b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 shutdown_connections 2024-03-25T17:36:38.627 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 1 -- 172.21.15.134:0/3971274627 wait complete. 2024-03-25T17:36:38.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:38.625+0000 7f2e139fb640 1 librados: shutdown 2024-03-25T17:36:38.639 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:45.445 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.560+0000 7f9bd3ae2640 10 monclient: get_monmap_and_config 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.560+0000 7f9bd3ae2640 10 monclient: build_initial_monmap 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.560+0000 7f9bd3ae2640 10 monclient: monmap: 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:36:45.561239+0000 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:36:45.561239+0000 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:45.561 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:36:45.562 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:36:45.562 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:45.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.561+0000 7f9bd3ae2640 1 Processor -- start 2024-03-25T17:36:45.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.561+0000 7f9bd3ae2640 1 -- start start 2024-03-25T17:36:45.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.561+0000 7f9bd3ae2640 10 monclient: init 2024-03-25T17:36:45.563 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f9bcc0b4d60 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f9bcc0b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 10 monclient(hunting): picked mon.noname-a con 0x7f9bcc14be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0af940 con 0x7f9bcc14be90 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0b0a70 con 0x7f9bcc0b4d60 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0b0d50 con 0x7f9bcc0b1730 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:45.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.562+0000 7f9bd3ae2640 10 monclient(hunting): authenticate will time out at 5758.825204s 2024-03-25T17:36:45.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.577+0000 7f9bd107d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f9bcc0b1730 0x7f9bcc0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:58240/0 (socket says 172.21.15.134:58240) 2024-03-25T17:36:45.578 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.577+0000 7f9bd107d640 1 -- 172.21.15.134:0/2728541737 learned_addr learned my addr 172.21.15.134:0/2728541737 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:45.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.595+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765450598 0 0) 0x7f9bcc0b0a70 con 0x7f9bcc0b4d60 2024-03-25T17:36:45.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.595+0000 7f9bd087c640 10 monclient(hunting): _negotiate 2024-03-25T17:36:45.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.595+0000 7f9bd087c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:45.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.595+0000 7f9bd087c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:45.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.595+0000 7f9bd087c640 10 monclient(hunting): my global_id is 4561 2024-03-25T17:36:45.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.595+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb8003520 con 0x7f9bcc0b4d60 2024-03-25T17:36:45.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.596+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569324531 0 0) 0x7f9bcc0b0d50 con 0x7f9bcc0b1730 2024-03-25T17:36:45.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.596+0000 7f9bd087c640 10 monclient(hunting): _negotiate 2024-03-25T17:36:45.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.596+0000 7f9bd087c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:45.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.596+0000 7f9bd087c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:45.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.596+0000 7f9bd087c640 10 monclient(hunting): my global_id is 34526 2024-03-25T17:36:45.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.596+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bcc0b0a70 con 0x7f9bcc0b1730 2024-03-25T17:36:45.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.599+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016222035 0 0) 0x7f9bcc0af940 con 0x7f9bcc14be90 2024-03-25T17:36:45.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.600+0000 7f9bd087c640 10 monclient(hunting): _negotiate 2024-03-25T17:36:45.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.600+0000 7f9bd087c640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:45.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.600+0000 7f9bd087c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:45.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.600+0000 7f9bd087c640 10 monclient(hunting): my global_id is 4536 2024-03-25T17:36:45.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.600+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bcc0b0d50 con 0x7f9bcc14be90 2024-03-25T17:36:45.607 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.606+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 279755733 0 0) 0x7f9bcc0b0a70 con 0x7f9bcc0b1730 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.606+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 >> v1:172.21.15.159:6789/0 conn(0x7f9bcc0b4d60 legacy=0x7f9bcc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.606+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 >> v1:172.21.15.134:6789/0 conn(0x7f9bcc14be90 legacy=0x7f9bcc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.606+0000 7f9bd087c640 10 monclient: _finish_hunting 0 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.606+0000 7f9bd087c640 1 monclient: found mon.noname-b 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.606+0000 7f9bd087c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc0b1030 con 0x7f9bcc0b1730 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 10 monclient: _finish_auth 0 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 10 monclient: _check_auth_tickets 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9bbc001670 con 0x7f9bcc0b1730 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd3ae2640 5 monclient: authenticate success, global_id 34526 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd3ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:45.608 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd087c640 10 monclient: dump: 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:45.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.607+0000 7f9bd3ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bbc0017d0 con 0x7f9bcc0b1730 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd087c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd087c640 1 -- 172.21.15.134:0/2728541737 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9bbc001be0 con 0x7f9bcc0b1730 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd087c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd087c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd087c640 10 monclient: dump: 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:45.615 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:45.616 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:45.616 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:45.616 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:45.616 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:45.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd3ae2640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:45.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.614+0000 7f9bd3ae2640 4 set_mon_vals no callback set 2024-03-25T17:36:45.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.615+0000 7f9bd3ae2640 10 monclient: shutdown 2024-03-25T17:36:45.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.615+0000 7f9bd3ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:45.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.615+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/2728541737 >> v1:172.21.15.134:6790/0 conn(0x7f9bcc0b1730 legacy=0x7f9bcc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:45.617 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.615+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/2728541737 shutdown_connections 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.615+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/2728541737 >> 172.21.15.134:0/2728541737 conn(0x7f9bcc0617a0 msgr2=0x7f9bcc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.615+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/2728541737 shutdown_connections 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.615+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/2728541737 wait complete. 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.616+0000 7f9bd3ae2640 10 monclient: build_initial_monmap 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.616+0000 7f9bd3ae2640 10 monclient: monmap: 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:36:45.618 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.616+0000 7f9bd3ae2640 1 librados: starting msgr at 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.616+0000 7f9bd3ae2640 1 librados: starting objecter 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 1 Processor -- start 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 1 -- start start 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 1 librados: setting wanted keys 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 1 librados: calling monclient init 2024-03-25T17:36:45.619 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient: init 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient(hunting): picked mon.noname-a con 0x7f9bcc0b1730 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f9bcc0b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f9bcc14be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc0a9610 con 0x7f9bcc0b1730 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc1433d0 con 0x7f9bcc14be90 2024-03-25T17:36:45.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bcc143560 con 0x7f9bcc0b4d60 2024-03-25T17:36:45.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:45.621 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.617+0000 7f9bd3ae2640 10 monclient(hunting): authenticate will time out at 5758.880633s 2024-03-25T17:36:45.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.631+0000 7f9bd207f640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f9bcc14be90 0x7f9bcc1e2210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:47128/0 (socket says 172.21.15.134:47128) 2024-03-25T17:36:45.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.631+0000 7f9bd207f640 1 -- 172.21.15.134:0/1663974882 learned_addr learned my addr 172.21.15.134:0/1663974882 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:45.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.634+0000 7f9bc27fc640 10 client.?.objecter ms_handle_connect 0x7f9bcc14be90 2024-03-25T17:36:45.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.634+0000 7f9bc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:45.637 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.637+0000 7f9bc27fc640 10 client.?.objecter ms_handle_connect 0x7f9bcc0b4d60 2024-03-25T17:36:45.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.637+0000 7f9bc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:45.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132753504 0 0) 0x7f9bcc1433d0 con 0x7f9bcc14be90 2024-03-25T17:36:45.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:45.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:45.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:45.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 10 monclient(hunting): my global_id is 4564 2024-03-25T17:36:45.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb0004eb0 con 0x7f9bcc14be90 2024-03-25T17:36:45.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 10 client.?.objecter ms_handle_connect 0x7f9bcc0b1730 2024-03-25T17:36:45.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.644+0000 7f9bc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:45.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.647+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750483800 0 0) 0x7f9bcc143560 con 0x7f9bcc0b4d60 2024-03-25T17:36:45.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.647+0000 7f9bc27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:45.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.647+0000 7f9bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:45.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.647+0000 7f9bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:45.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.647+0000 7f9bc27fc640 10 monclient(hunting): my global_id is 34529 2024-03-25T17:36:45.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.647+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bcc1433d0 con 0x7f9bcc0b4d60 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.660+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1141571008 0 0) 0x7f9bb0004eb0 con 0x7f9bcc14be90 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.660+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bcc143560 con 0x7f9bcc14be90 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.660+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9bc4001c70 con 0x7f9bcc14be90 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.660+0000 7f9bc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.660+0000 7f9bc27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.660+0000 7f9bc27fc640 10 monclient(hunting): dump: 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:45.661 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:45.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.660+0000 7f9bc27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:45.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.662+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1203281705 0 0) 0x7f9bcc0a9610 con 0x7f9bcc0b1730 2024-03-25T17:36:45.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.662+0000 7f9bc27fc640 10 monclient(hunting): _negotiate 2024-03-25T17:36:45.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.662+0000 7f9bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:45.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.662+0000 7f9bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:45.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.662+0000 7f9bc27fc640 10 monclient(hunting): my global_id is 4539 2024-03-25T17:36:45.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.662+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bb0004eb0 con 0x7f9bcc0b1730 2024-03-25T17:36:45.674 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 689561588 0 0) 0x7f9bcc143560 con 0x7f9bcc14be90 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 >> v1:172.21.15.134:6790/0 conn(0x7f9bcc0b4d60 legacy=0x7f9bcc1d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 >> v1:172.21.15.134:6789/0 conn(0x7f9bcc0b1730 legacy=0x7f9bcc1d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 10 monclient: _finish_hunting 0 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 1 monclient: found mon.b 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bcc1436f0 con 0x7f9bcc14be90 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 10 monclient: _finish_auth 0 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 10 monclient: _check_auth_tickets 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 5 monclient: authenticate success, global_id 4564 2024-03-25T17:36:45.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 10 monclient: _renew_subs 2024-03-25T17:36:45.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:45.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bcc1429a0 con 0x7f9bcc14be90 2024-03-25T17:36:45.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:45.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 10 monclient: _renew_subs 2024-03-25T17:36:45.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:45.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bcc140560 con 0x7f9bcc14be90 2024-03-25T17:36:45.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.673+0000 7f9bd3ae2640 1 librados: init done 2024-03-25T17:36:45.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.674+0000 7f9bd3ae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:45.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.674+0000 7f9bd3ae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:45.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.674+0000 7f9bd3ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:45.677 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.674+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bcc142ca0 con 0x7f9bcc14be90 2024-03-25T17:36:45.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.945+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bc4003990 con 0x7f9bcc14be90 2024-03-25T17:36:45.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.945+0000 7f9bc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.945+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9bc4003da0 con 0x7f9bcc14be90 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.945+0000 7f9bc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.945+0000 7f9bc27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.945+0000 7f9bc27fc640 10 monclient: dump: 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:45.947 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:45.948 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:45.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.945+0000 7f9bc3fff640 4 set_mon_vals no callback set 2024-03-25T17:36:45.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.946+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f9bc4003fc0 con 0x7f9bcc14be90 2024-03-25T17:36:45.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.947+0000 7f9bd187e640 10 monclient: get_auth_request con 0x7f9bb006de00 auth_method 0 2024-03-25T17:36:45.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.947+0000 7f9bc27fc640 10 client.4564.objecter ms_handle_connect 0x7f9bb006de00 2024-03-25T17:36:45.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.951+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f9bc4054a30 con 0x7f9bcc14be90 2024-03-25T17:36:45.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.951+0000 7f9bc27fc640 10 client.4564.objecter ms_dispatch 0x7f9bcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:45.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.951+0000 7f9bc27fc640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:45.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.951+0000 7f9bc27fc640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:45.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.951+0000 7f9bc27fc640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T17:36:45.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.952+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9bc4050540 con 0x7f9bcc14be90 2024-03-25T17:36:45.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.952+0000 7f9bc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:45.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:45.952+0000 7f9bc27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:46.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.213+0000 7f9bd3ae2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:46.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.213+0000 7f9bd3ae2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:46.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.213+0000 7f9bd3ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:46.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.213+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bcc0a97a0 con 0x7f9bcc14be90 2024-03-25T17:36:46.240 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.240+0000 7f9bc27fc640 1 -- 172.21.15.134:0/1663974882 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f9bc4052470 con 0x7f9bcc14be90 2024-03-25T17:36:46.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.240+0000 7f9bc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:46.241 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.240+0000 7f9bc27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:46.241 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:46.241 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:46.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 10 librados: watch_flush enter 2024-03-25T17:36:46.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 10 librados: watch_flush exit 2024-03-25T17:36:46.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T17:36:46.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 10 client.4564.objecter successfully canceled tick 2024-03-25T17:36:46.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 >> v1:172.21.15.134:6800/610837766 conn(0x7f9bb006de00 legacy=0x7f9bb00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:46.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 10 monclient: shutdown 2024-03-25T17:36:46.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:46.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 >> v1:172.21.15.159:6789/0 conn(0x7f9bcc14be90 legacy=0x7f9bcc1e2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:46.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 shutdown_connections 2024-03-25T17:36:46.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 >> 172.21.15.134:0/1663974882 conn(0x7f9bcc0617a0 msgr2=0x7f9bcc13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:46.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 shutdown_connections 2024-03-25T17:36:46.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 1 -- 172.21.15.134:0/1663974882 wait complete. 2024-03-25T17:36:46.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:46.242+0000 7f9bd3ae2640 1 librados: shutdown 2024-03-25T17:36:46.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:36:50.293 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:55.309 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.340 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:55.369 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.341 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:55.370 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.352 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:07:55.381 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.354 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:55.370 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.401 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:07:55.418 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.402 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:55.419 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.407 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:55.423 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:50.442 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:07:55.458 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:36:53.062 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:36:53.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.183+0000 7f6dd6396640 10 monclient: get_monmap_and_config 2024-03-25T17:36:53.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.183+0000 7f6dd6396640 10 monclient: build_initial_monmap 2024-03-25T17:36:53.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.183+0000 7f6dd6396640 10 monclient: monmap: 2024-03-25T17:36:53.184 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:53.184 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:53.184 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:36:53.184763+0000 2024-03-25T17:36:53.184 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:36:53.184763+0000 2024-03-25T17:36:53.185 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:53.185 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:53.185 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:53.185 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:36:53.185 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:36:53.185 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:53.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.184+0000 7f6dd6396640 1 Processor -- start 2024-03-25T17:36:53.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.184+0000 7f6dd6396640 1 -- start start 2024-03-25T17:36:53.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.184+0000 7f6dd6396640 10 monclient: init 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 10 monclient(hunting): picked mon.noname-c con 0x7f6dd00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 10 monclient(hunting): picked mon.noname-a con 0x7f6dd013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 10 monclient(hunting): picked mon.noname-b con 0x7f6dd01430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd0142520 con 0x7f6dd013f710 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd00b28a0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd00b2a30 con 0x7f6dd01430c0 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:53.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.185+0000 7f6dd6396640 10 monclient(hunting): authenticate will time out at 5766.448343s 2024-03-25T17:36:53.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.200+0000 7f6dd4933640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f6dd01430c0 0x7f6dd014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46722/0 (socket says 172.21.15.134:46722) 2024-03-25T17:36:53.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.200+0000 7f6dd4933640 1 -- 172.21.15.134:0/2984500155 learned_addr learned my addr 172.21.15.134:0/2984500155 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:53.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.216+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403993735 0 0) 0x7f6dd00b2a30 con 0x7f6dd01430c0 2024-03-25T17:36:53.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.216+0000 7f6dceffd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:53.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.216+0000 7f6dceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:53.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.216+0000 7f6dceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:53.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.216+0000 7f6dceffd640 10 monclient(hunting): my global_id is 34532 2024-03-25T17:36:53.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.216+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db8003520 con 0x7f6dd01430c0 2024-03-25T17:36:53.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.224+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483242612 0 0) 0x7f6dd00b28a0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.224+0000 7f6dceffd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:53.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.224+0000 7f6dceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:53.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.224+0000 7f6dceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:53.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.224+0000 7f6dceffd640 10 monclient(hunting): my global_id is 4567 2024-03-25T17:36:53.225 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.224+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd00b2a30 con 0x7f6dd00a98f0 2024-03-25T17:36:53.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.230+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516631768 0 0) 0x7f6dd0142520 con 0x7f6dd013f710 2024-03-25T17:36:53.230 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.230+0000 7f6dceffd640 10 monclient(hunting): _negotiate 2024-03-25T17:36:53.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.230+0000 7f6dceffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:53.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.230+0000 7f6dceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:53.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.230+0000 7f6dceffd640 10 monclient(hunting): my global_id is 4542 2024-03-25T17:36:53.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.230+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd00b28a0 con 0x7f6dd013f710 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3169062022 0 0) 0x7f6dd00b2a30 con 0x7f6dd00a98f0 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 >> v1:172.21.15.134:6790/0 conn(0x7f6dd01430c0 legacy=0x7f6dd014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 >> v1:172.21.15.134:6789/0 conn(0x7f6dd013f710 legacy=0x7f6dd01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 10 monclient: _finish_hunting 0 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 1 monclient: found mon.noname-c 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd00b44c0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 10 monclient: _finish_auth 0 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 10 monclient: _check_auth_tickets 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:53.236 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6dc0001670 con 0x7f6dd00a98f0 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dceffd640 10 monclient: dump: 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dd6396640 5 monclient: authenticate success, global_id 4567 2024-03-25T17:36:53.237 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.235+0000 7f6dd6396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:36:53.247 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc00017d0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dceffd640 1 -- 172.21.15.134:0/2984500155 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6dc0001be0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dceffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dceffd640 10 monclient: dump: 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:53.248 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:53.249 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:53.249 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:53.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dd6396640 10 monclient: get_monmap_and_config success 2024-03-25T17:36:53.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.247+0000 7f6dd6396640 4 set_mon_vals no callback set 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.248+0000 7f6dd6396640 10 monclient: shutdown 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.248+0000 7f6dd6396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.248+0000 7f6dd6396640 1 -- 172.21.15.134:0/2984500155 >> v1:172.21.15.159:6789/0 conn(0x7f6dd00a98f0 legacy=0x7f6dd00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.248+0000 7f6dd6396640 1 -- 172.21.15.134:0/2984500155 shutdown_connections 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.248+0000 7f6dd6396640 1 -- 172.21.15.134:0/2984500155 >> 172.21.15.134:0/2984500155 conn(0x7f6dd00b7ea0 msgr2=0x7f6dd00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.248+0000 7f6dd6396640 1 -- 172.21.15.134:0/2984500155 shutdown_connections 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.248+0000 7f6dd6396640 1 -- 172.21.15.134:0/2984500155 wait complete. 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.249+0000 7f6dd6396640 10 monclient: build_initial_monmap 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.249+0000 7f6dd6396640 10 monclient: monmap: 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:36:53.250 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:36:53.251 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:36:53.251 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:36:53.251 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:36:53.251 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:36:53.251 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:36:53.251 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:36:53.251 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:53.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.250+0000 7f6dd6396640 1 librados: starting msgr at 2024-03-25T17:36:53.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.250+0000 7f6dd6396640 1 librados: starting objecter 2024-03-25T17:36:53.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.250+0000 7f6dd6396640 1 Processor -- start 2024-03-25T17:36:53.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.250+0000 7f6dd6396640 1 -- start start 2024-03-25T17:36:53.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.250+0000 7f6dd6396640 1 librados: setting wanted keys 2024-03-25T17:36:53.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.250+0000 7f6dd6396640 1 librados: calling monclient init 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.250+0000 7f6dd6396640 10 monclient: init 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 10 monclient: _reopen_session rank -1 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 10 monclient(hunting): picked mon.noname-c con 0x7f6dd00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 10 monclient(hunting): picked mon.noname-a con 0x7f6dd013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 10 monclient(hunting): picked mon.noname-b con 0x7f6dd01430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd01399f0 con 0x7f6dd013f710 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd013adc0 con 0x7f6dd01430c0 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd013b0a0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 10 monclient(hunting): _renew_subs 2024-03-25T17:36:53.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.251+0000 7f6dd6396640 10 monclient(hunting): authenticate will time out at 5766.514573s 2024-03-25T17:36:53.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.266+0000 7f6dcffff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f6dd00a98f0 0x7f6dd00b76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46770/0 (socket says 172.21.15.134:46770) 2024-03-25T17:36:53.266 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.266+0000 7f6dcffff640 1 -- 172.21.15.134:0/2501945220 learned_addr learned my addr 172.21.15.134:0/2501945220 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:36:53.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.270+0000 7f6dccff9640 10 client.?.objecter ms_handle_connect 0x7f6dd01430c0 2024-03-25T17:36:53.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.270+0000 7f6dccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:53.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.272+0000 7f6dccff9640 10 client.?.objecter ms_handle_connect 0x7f6dd00a98f0 2024-03-25T17:36:53.272 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.272+0000 7f6dccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:53.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.278+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881487849 0 0) 0x7f6dd013adc0 con 0x7f6dd01430c0 2024-03-25T17:36:53.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.278+0000 7f6dccff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:53.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.278+0000 7f6dccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:53.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.278+0000 7f6dccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:53.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.278+0000 7f6dccff9640 10 monclient(hunting): my global_id is 4570 2024-03-25T17:36:53.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.278+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dbc004eb0 con 0x7f6dd01430c0 2024-03-25T17:36:53.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.279+0000 7f6dccff9640 10 client.?.objecter ms_handle_connect 0x7f6dd013f710 2024-03-25T17:36:53.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.279+0000 7f6dccff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:36:53.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.280+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599598212 0 0) 0x7f6dd013b0a0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.280+0000 7f6dccff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:53.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.280+0000 7f6dccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:53.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.280+0000 7f6dccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:53.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.280+0000 7f6dccff9640 10 monclient(hunting): my global_id is 34535 2024-03-25T17:36:53.281 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.280+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd013adc0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.291+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997241848 0 0) 0x7f6dd01399f0 con 0x7f6dd013f710 2024-03-25T17:36:53.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.291+0000 7f6dccff9640 10 monclient(hunting): _negotiate 2024-03-25T17:36:53.291 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.291+0000 7f6dccff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:36:53.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.291+0000 7f6dccff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:36:53.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.291+0000 7f6dccff9640 10 monclient(hunting): my global_id is 4545 2024-03-25T17:36:53.292 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.291+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6dd013b0a0 con 0x7f6dd013f710 2024-03-25T17:36:53.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.295+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395409664 0 0) 0x7f6dbc004eb0 con 0x7f6dd01430c0 2024-03-25T17:36:53.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.296+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dd01399f0 con 0x7f6dd01430c0 2024-03-25T17:36:53.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.296+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6dc8001af0 con 0x7f6dd01430c0 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.296+0000 7f6dccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.296+0000 7f6dccff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.296+0000 7f6dccff9640 10 monclient(hunting): dump: 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:53.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.296+0000 7f6dccff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:36:53.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.300+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4135415923 0 0) 0x7f6dd013adc0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.300+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6dbc004eb0 con 0x7f6dd00a98f0 2024-03-25T17:36:53.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.300+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6dc0001670 con 0x7f6dd00a98f0 2024-03-25T17:36:53.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.300+0000 7f6dccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:53.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.300+0000 7f6dccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.300+0000 7f6dccff9640 10 monclient(hunting): dump: 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:53.302 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:53.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 854881625 0 0) 0x7f6dd01399f0 con 0x7f6dd01430c0 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 >> v1:172.21.15.134:6790/0 conn(0x7f6dd00a98f0 legacy=0x7f6dd00b76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 >> v1:172.21.15.134:6789/0 conn(0x7f6dd013f710 legacy=0x7f6dd01daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 10 monclient: _finish_hunting 0 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 1 monclient: found mon.b 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd00b6ea0 con 0x7f6dd01430c0 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 10 monclient: _finish_auth 0 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 10 monclient: _check_auth_tickets 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 5 monclient: authenticate success, global_id 4570 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 10 monclient: _renew_subs 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd013b7a0 con 0x7f6dd01430c0 2024-03-25T17:36:53.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:36:53.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 10 monclient: _renew_subs 2024-03-25T17:36:53.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:53.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd0139740 con 0x7f6dd01430c0 2024-03-25T17:36:53.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.311+0000 7f6dd6396640 1 librados: init done 2024-03-25T17:36:53.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.312+0000 7f6dd6396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:53.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.312+0000 7f6dd6396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:53.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.312+0000 7f6dd6396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:53.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.312+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd00b3ef0 con 0x7f6dd01430c0 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.593+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6dc8001f20 con 0x7f6dd01430c0 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.593+0000 7f6dccff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.593+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f6dc8004d00 con 0x7f6dd01430c0 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.593+0000 7f6dccff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.593+0000 7f6dccff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.593+0000 7f6dccff9640 10 monclient: dump: 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:36:53.594 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:36:53.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.593+0000 7f6dce7fc640 4 set_mon_vals no callback set 2024-03-25T17:36:53.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.594+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f6dc801acd0 con 0x7f6dd01430c0 2024-03-25T17:36:53.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.594+0000 7f6dcffff640 10 monclient: get_auth_request con 0x7f6dbc06df20 auth_method 0 2024-03-25T17:36:53.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.595+0000 7f6dccff9640 10 client.4570.objecter ms_handle_connect 0x7f6dbc06df20 2024-03-25T17:36:53.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.599+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f6dc8054990 con 0x7f6dd01430c0 2024-03-25T17:36:53.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.599+0000 7f6dccff9640 10 client.4570.objecter ms_dispatch 0x7f6dd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:36:53.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.599+0000 7f6dccff9640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:36:53.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.599+0000 7f6dccff9640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:36:53.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.599+0000 7f6dccff9640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T17:36:53.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.600+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6dc80504a0 con 0x7f6dd01430c0 2024-03-25T17:36:53.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.601+0000 7f6dccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:36:53.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.601+0000 7f6dccff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:36:53.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.821+0000 7f6dd6396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:53.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.821+0000 7f6dd6396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:53.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.821+0000 7f6dd6396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:36:53.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.821+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dd00627b0 con 0x7f6dd01430c0 2024-03-25T17:36:53.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.843+0000 7f6dccff9640 1 -- 172.21.15.134:0/2501945220 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f6dc80523d0 con 0x7f6dd01430c0 2024-03-25T17:36:53.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.843+0000 7f6dccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:36:53.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.843+0000 7f6dccff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:36:53.844 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:36:53.844 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:36:53.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 10 librados: watch_flush enter 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 10 librados: watch_flush exit 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 10 client.4570.objecter successfully canceled tick 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 >> v1:172.21.15.134:6800/610837766 conn(0x7f6dbc06df20 legacy=0x7f6dbc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 10 monclient: shutdown 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 >> v1:172.21.15.159:6789/0 conn(0x7f6dd01430c0 legacy=0x7f6dd01db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.845+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 shutdown_connections 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.846+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 >> 172.21.15.134:0/2501945220 conn(0x7f6dd00b7ea0 msgr2=0x7f6dd0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.846+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 shutdown_connections 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.846+0000 7f6dd6396640 1 -- 172.21.15.134:0/2501945220 wait complete. 2024-03-25T17:36:53.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:36:53.846+0000 7f6dd6396640 1 librados: shutdown 2024-03-25T17:36:53.858 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:00.665 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:00.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.778+0000 7f81f19c6640 10 monclient: get_monmap_and_config 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.778+0000 7f81f19c6640 10 monclient: build_initial_monmap 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.778+0000 7f81f19c6640 10 monclient: monmap: 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:00.779410+0000 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:00.779410+0000 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:00.779 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:00.780 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:00.780 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:00.780 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:00.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 1 Processor -- start 2024-03-25T17:37:00.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 1 -- start start 2024-03-25T17:37:00.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient: init 2024-03-25T17:37:00.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient(hunting): picked mon.noname-b con 0x7f81ec06ab80 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient(hunting): picked mon.noname-a con 0x7f81ec06af60 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient(hunting): picked mon.noname-c con 0x7f81ec0b00b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec0b0500 con 0x7f81ec06af60 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec13ffa0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec0af520 con 0x7f81ec06ab80 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:00.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.779+0000 7f81f19c6640 10 monclient(hunting): authenticate will time out at 5774.042688s 2024-03-25T17:37:00.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.790+0000 7f81eb7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f81ec0b00b0 0x7f81ec14dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34030/0 (socket says 172.21.15.134:34030) 2024-03-25T17:37:00.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.790+0000 7f81eb7fe640 1 -- 172.21.15.134:0/2017804106 learned_addr learned my addr 172.21.15.134:0/2017804106 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:00.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.798+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516276864 0 0) 0x7f81ec13ffa0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.798+0000 7f81e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:00.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.798+0000 7f81e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:00.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.798+0000 7f81e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:00.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.798+0000 7f81e9ffb640 10 monclient(hunting): my global_id is 4573 2024-03-25T17:37:00.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.798+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d4003520 con 0x7f81ec0b00b0 2024-03-25T17:37:00.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2899724811 0 0) 0x7f81d4003520 con 0x7f81ec0b00b0 2024-03-25T17:37:00.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 >> v1:172.21.15.134:6790/0 conn(0x7f81ec06ab80 legacy=0x7f81ec0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:37:00.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 >> v1:172.21.15.134:6789/0 conn(0x7f81ec06af60 legacy=0x7f81ec13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:37:00.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 1 monclient: found mon.noname-c 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec072890 con 0x7f81ec0b00b0 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 10 monclient: _finish_auth 0 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f81e4001a10 con 0x7f81ec0b00b0 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.806+0000 7f81e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.807+0000 7f81e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.807+0000 7f81e9ffb640 10 monclient: dump: 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:00.808 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.807+0000 7f81f19c6640 5 monclient: authenticate success, global_id 4573 2024-03-25T17:37:00.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.807+0000 7f81f19c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:00.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81e4001be0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81e9ffb640 1 -- 172.21.15.134:0/2017804106 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f81e4002030 con 0x7f81ec0b00b0 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81f19c6640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81f19c6640 4 set_mon_vals no callback set 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81e9ffb640 10 monclient: dump: 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:00.814 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:00.815 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:00.815 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:00.815 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:00.815 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:00.815 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:00.815 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:00.817 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.813+0000 7f81f19c6640 10 monclient: shutdown 2024-03-25T17:37:00.817 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:00.817 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 1 -- 172.21.15.134:0/2017804106 >> v1:172.21.15.159:6789/0 conn(0x7f81ec0b00b0 legacy=0x7f81ec14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 1 -- 172.21.15.134:0/2017804106 shutdown_connections 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 1 -- 172.21.15.134:0/2017804106 >> 172.21.15.134:0/2017804106 conn(0x7f81ec0b4a80 msgr2=0x7f81ec0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 1 -- 172.21.15.134:0/2017804106 shutdown_connections 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 1 -- 172.21.15.134:0/2017804106 wait complete. 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 10 monclient: build_initial_monmap 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.814+0000 7f81f19c6640 10 monclient: monmap: 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:00.818 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 1 librados: starting msgr at 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 1 librados: starting objecter 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 1 Processor -- start 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 1 -- start start 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 1 librados: setting wanted keys 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 1 librados: calling monclient init 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 10 monclient: init 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:00.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.815+0000 7f81f19c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 10 monclient(hunting): picked mon.noname-a con 0x7f81ec06ab80 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 10 monclient(hunting): picked mon.noname-b con 0x7f81ec0b00b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 10 monclient(hunting): picked mon.noname-c con 0x7f81ec143f90 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec143460 con 0x7f81ec06ab80 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec144fd0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ec1da890 con 0x7f81ec143f90 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:00.820 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.816+0000 7f81f19c6640 10 monclient(hunting): authenticate will time out at 5774.078843s 2024-03-25T17:37:00.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.821+0000 7f81ea7fc640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f81ec0b00b0 0x7f81ec145da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34066/0 (socket says 172.21.15.134:34066) 2024-03-25T17:37:00.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.821+0000 7f81ea7fc640 1 -- 172.21.15.134:0/1857314580 learned_addr learned my addr 172.21.15.134:0/1857314580 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:00.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.821+0000 7f81d3fff640 10 client.?.objecter ms_handle_connect 0x7f81ec0b00b0 2024-03-25T17:37:00.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.821+0000 7f81d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:00.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.826+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472519103 0 0) 0x7f81ec144fd0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.826+0000 7f81d3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:00.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.826+0000 7f81d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:00.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.826+0000 7f81d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:00.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.826+0000 7f81d3fff640 10 monclient(hunting): my global_id is 4576 2024-03-25T17:37:00.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.826+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d8004eb0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.834+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3906434585 0 0) 0x7f81d8004eb0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.835 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.835+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81ec144fd0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.835+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f81e0001960 con 0x7f81ec0b00b0 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.835+0000 7f81d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.835+0000 7f81d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.835+0000 7f81d3fff640 10 monclient(hunting): dump: 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:00.836 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:00.837 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:00.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.835+0000 7f81d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:00.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2214661419 0 0) 0x7f81ec144fd0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 >> v1:172.21.15.134:6790/0 conn(0x7f81ec143f90 legacy=0x7f81ec1e2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:00.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 >> v1:172.21.15.134:6789/0 conn(0x7f81ec06ab80 legacy=0x7f81ec1dadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:00.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:37:00.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 1 monclient: found mon.b 2024-03-25T17:37:00.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ec143bb0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 10 monclient: _finish_auth 0 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 5 monclient: authenticate success, global_id 4576 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 10 monclient: _renew_subs 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ec1db4c0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 10 monclient: _renew_subs 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ec1458b0 con 0x7f81ec0b00b0 2024-03-25T17:37:00.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.843+0000 7f81f19c6640 1 librados: init done 2024-03-25T17:37:00.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.844+0000 7f81f19c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:00.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.844+0000 7f81f19c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:00.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.844+0000 7f81f19c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:00.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:00.844+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ec0627b0 con 0x7f81ec0b00b0 2024-03-25T17:37:01.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.122+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81e0001ac0 con 0x7f81ec0b00b0 2024-03-25T17:37:01.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.122+0000 7f81d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:01.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.122+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f81e0001d80 con 0x7f81ec0b00b0 2024-03-25T17:37:01.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.122+0000 7f81d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:01.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.122+0000 7f81d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:01.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.122+0000 7f81d3fff640 10 monclient: dump: 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:01.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.122+0000 7f81e97fa640 4 set_mon_vals no callback set 2024-03-25T17:37:01.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.123+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f81e0003b00 con 0x7f81ec0b00b0 2024-03-25T17:37:01.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.124+0000 7f81eaffd640 10 monclient: get_auth_request con 0x7f81d806dbc0 auth_method 0 2024-03-25T17:37:01.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.124+0000 7f81d3fff640 10 client.4576.objecter ms_handle_connect 0x7f81d806dbc0 2024-03-25T17:37:01.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.127+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f81e0054860 con 0x7f81ec0b00b0 2024-03-25T17:37:01.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.127+0000 7f81d3fff640 10 client.4576.objecter ms_dispatch 0x7f81ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:01.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.127+0000 7f81d3fff640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:01.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.127+0000 7f81d3fff640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:01.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.127+0000 7f81d3fff640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T17:37:01.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.128+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81e0050370 con 0x7f81ec0b00b0 2024-03-25T17:37:01.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.128+0000 7f81d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:01.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.128+0000 7f81d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:01.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.336+0000 7f81f19c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:01.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.337+0000 7f81f19c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:01.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.337+0000 7f81f19c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:01.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.337+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81ec0adf60 con 0x7f81ec0b00b0 2024-03-25T17:37:01.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.373+0000 7f81d3fff640 1 -- 172.21.15.134:0/1857314580 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f81e00522a0 con 0x7f81ec0b00b0 2024-03-25T17:37:01.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.373+0000 7f81d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:01.374 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.373+0000 7f81d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:01.374 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:01.374 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:01.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 10 librados: watch_flush enter 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 10 librados: watch_flush exit 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 10 client.4576.objecter successfully canceled tick 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 >> v1:172.21.15.134:6800/610837766 conn(0x7f81d806dbc0 legacy=0x7f81d8070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 10 monclient: shutdown 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 >> v1:172.21.15.159:6789/0 conn(0x7f81ec0b00b0 legacy=0x7f81ec145da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 shutdown_connections 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.375+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 >> 172.21.15.134:0/1857314580 conn(0x7f81ec0b4a80 msgr2=0x7f81ec0701d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.376+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 shutdown_connections 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.376+0000 7f81f19c6640 1 -- 172.21.15.134:0/1857314580 wait complete. 2024-03-25T17:37:01.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:01.376+0000 7f81f19c6640 1 librados: shutdown 2024-03-25T17:37:01.388 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:08.194 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:08.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.312+0000 7ff16cb2b640 10 monclient: get_monmap_and_config 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.312+0000 7ff16cb2b640 10 monclient: build_initial_monmap 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.312+0000 7ff16cb2b640 10 monclient: monmap: 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:08.313282+0000 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:08.313282+0000 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:08.313 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.312+0000 7ff16cb2b640 1 Processor -- start 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 1 -- start start 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient: init 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient(hunting): picked mon.noname-b con 0x7ff1680b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient(hunting): picked mon.noname-c con 0x7ff1680b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient(hunting): picked mon.noname-a con 0x7ff16814be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1680af940 con 0x7ff16814be90 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1680b0a70 con 0x7ff1680b1730 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1680b0d50 con 0x7ff1680b4d60 2024-03-25T17:37:08.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:08.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.313+0000 7ff16cb2b640 10 monclient(hunting): authenticate will time out at 5781.576718s 2024-03-25T17:37:08.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.319+0000 7ff166d9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7ff16814be90 0x7ff16814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:50208/0 (socket says 172.21.15.134:50208) 2024-03-25T17:37:08.320 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.319+0000 7ff166d9d640 1 -- 172.21.15.134:0/3939352290 learned_addr learned my addr 172.21.15.134:0/3939352290 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:08.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.331+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731615121 0 0) 0x7ff1680af940 con 0x7ff16814be90 2024-03-25T17:37:08.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.331+0000 7ff16559a640 10 monclient(hunting): _negotiate 2024-03-25T17:37:08.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.331+0000 7ff16559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:08.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.331+0000 7ff16559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:08.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.331+0000 7ff16559a640 10 monclient(hunting): my global_id is 4548 2024-03-25T17:37:08.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.331+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff150003520 con 0x7ff16814be90 2024-03-25T17:37:08.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1737462008 0 0) 0x7ff150003520 con 0x7ff16814be90 2024-03-25T17:37:08.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 >> v1:172.21.15.134:6790/0 conn(0x7ff1680b4d60 legacy=0x7ff16813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:37:08.340 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 >> v1:172.21.15.159:6789/0 conn(0x7ff1680b1730 legacy=0x7ff1680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 10 monclient: _finish_hunting 0 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 1 monclient: found mon.noname-a 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1680b1030 con 0x7ff16814be90 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 10 monclient: _finish_auth 0 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 10 monclient: _check_auth_tickets 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff160001a10 con 0x7ff16814be90 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16559a640 10 monclient: dump: 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:08.341 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16cb2b640 5 monclient: authenticate success, global_id 4548 2024-03-25T17:37:08.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.339+0000 7ff16cb2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff160001be0 con 0x7ff16814be90 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16559a640 1 -- 172.21.15.134:0/3939352290 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff160001ff0 con 0x7ff16814be90 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16559a640 10 monclient: dump: 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:08.355 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16cb2b640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:08.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.354+0000 7ff16cb2b640 4 set_mon_vals no callback set 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.355+0000 7ff16cb2b640 10 monclient: shutdown 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.355+0000 7ff16cb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.355+0000 7ff16cb2b640 1 -- 172.21.15.134:0/3939352290 >> v1:172.21.15.134:6789/0 conn(0x7ff16814be90 legacy=0x7ff16814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.355+0000 7ff16cb2b640 1 -- 172.21.15.134:0/3939352290 shutdown_connections 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.355+0000 7ff16cb2b640 1 -- 172.21.15.134:0/3939352290 >> 172.21.15.134:0/3939352290 conn(0x7ff1680617a0 msgr2=0x7ff168061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.355+0000 7ff16cb2b640 1 -- 172.21.15.134:0/3939352290 shutdown_connections 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.355+0000 7ff16cb2b640 1 -- 172.21.15.134:0/3939352290 wait complete. 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.356+0000 7ff16cb2b640 10 monclient: build_initial_monmap 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.356+0000 7ff16cb2b640 10 monclient: monmap: 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:08.358 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 librados: starting msgr at 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 librados: starting objecter 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 Processor -- start 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 -- start start 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 librados: setting wanted keys 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 librados: calling monclient init 2024-03-25T17:37:08.359 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient: init 2024-03-25T17:37:08.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:08.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:37:08.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient(hunting): picked mon.noname-b con 0x7ff1680b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:08.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient(hunting): picked mon.noname-a con 0x7ff1680b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:08.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient(hunting): picked mon.noname-c con 0x7ff16814be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:08.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1681daf60 con 0x7ff1680b4d60 2024-03-25T17:37:08.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1681422d0 con 0x7ff1680b1730 2024-03-25T17:37:08.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff168140a90 con 0x7ff16814be90 2024-03-25T17:37:08.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:08.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.357+0000 7ff16cb2b640 10 monclient(hunting): authenticate will time out at 5781.620756s 2024-03-25T17:37:08.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.370+0000 7ff16659c640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff1680b1730 0x7ff168140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:38724/0 (socket says 172.21.15.134:38724) 2024-03-25T17:37:08.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.370+0000 7ff16659c640 1 -- 172.21.15.134:0/77212163 learned_addr learned my addr 172.21.15.134:0/77212163 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:08.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.372+0000 7ff156ffd640 10 client.?.objecter ms_handle_connect 0x7ff1680b4d60 2024-03-25T17:37:08.373 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.372+0000 7ff156ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:08.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.376+0000 7ff156ffd640 10 client.?.objecter ms_handle_connect 0x7ff1680b1730 2024-03-25T17:37:08.376 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.376+0000 7ff156ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:08.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.377+0000 7ff156ffd640 10 client.?.objecter ms_handle_connect 0x7ff16814be90 2024-03-25T17:37:08.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.377+0000 7ff156ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:08.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.378+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694310164 0 0) 0x7ff1681daf60 con 0x7ff1680b4d60 2024-03-25T17:37:08.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.378+0000 7ff156ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:37:08.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.378+0000 7ff156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:08.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.378+0000 7ff156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:08.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.378+0000 7ff156ffd640 10 monclient(hunting): my global_id is 4551 2024-03-25T17:37:08.379 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.378+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff148004eb0 con 0x7ff1680b4d60 2024-03-25T17:37:08.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.385+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316914571 0 0) 0x7ff168140a90 con 0x7ff16814be90 2024-03-25T17:37:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.385+0000 7ff156ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:37:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.385+0000 7ff156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.385+0000 7ff156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.385+0000 7ff156ffd640 10 monclient(hunting): my global_id is 34538 2024-03-25T17:37:08.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.385+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1681daf60 con 0x7ff16814be90 2024-03-25T17:37:08.388 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.388+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795312039 0 0) 0x7ff1681422d0 con 0x7ff1680b1730 2024-03-25T17:37:08.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.388+0000 7ff156ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:37:08.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.388+0000 7ff156ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:08.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.388+0000 7ff156ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:08.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.388+0000 7ff156ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T17:37:08.389 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.388+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff168140a90 con 0x7ff1680b1730 2024-03-25T17:37:08.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.391+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4100618642 0 0) 0x7ff148004eb0 con 0x7ff1680b4d60 2024-03-25T17:37:08.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.391+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1681422d0 con 0x7ff1680b4d60 2024-03-25T17:37:08.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.391+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff15c001930 con 0x7ff1680b4d60 2024-03-25T17:37:08.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.391+0000 7ff156ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:08.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.391+0000 7ff156ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:08.392 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.391+0000 7ff156ffd640 10 monclient(hunting): dump: 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.391+0000 7ff156ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.397+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2043399889 0 0) 0x7ff1681daf60 con 0x7ff16814be90 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.397+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff148004eb0 con 0x7ff16814be90 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.397+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff160001ef0 con 0x7ff16814be90 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.397+0000 7ff156ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.397+0000 7ff156ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.397+0000 7ff156ffd640 10 monclient(hunting): dump: 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:08.398 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.399+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4052731442 0 0) 0x7ff168140a90 con 0x7ff1680b1730 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.399+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1681daf60 con 0x7ff1680b1730 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.399+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff1580018a0 con 0x7ff1680b1730 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.399+0000 7ff156ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.399+0000 7ff156ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.399+0000 7ff156ffd640 10 monclient(hunting): dump: 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:08.400 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:08.401 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:08.401 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 923475410 0 0) 0x7ff1681422d0 con 0x7ff1680b4d60 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 >> v1:172.21.15.134:6790/0 conn(0x7ff16814be90 legacy=0x7ff1681d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 >> v1:172.21.15.159:6789/0 conn(0x7ff1680b1730 legacy=0x7ff168140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 1 monclient: found mon.a 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:08.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1681d79e0 con 0x7ff1680b4d60 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 10 monclient: _finish_auth 0 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff156ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 5 monclient: authenticate success, global_id 4551 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 10 monclient: _renew_subs 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1681dbb00 con 0x7ff1680b4d60 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 10 monclient: _renew_subs 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1681d85c0 con 0x7ff1680b4d60 2024-03-25T17:37:08.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.407+0000 7ff16cb2b640 1 librados: init done 2024-03-25T17:37:08.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.408+0000 7ff16cb2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:08.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.408+0000 7ff16cb2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:08.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.408+0000 7ff16cb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:08.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.408+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff168140c20 con 0x7ff1680b4d60 2024-03-25T17:37:08.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.414+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff15c0035f0 con 0x7ff1680b4d60 2024-03-25T17:37:08.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.414+0000 7ff156ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:08.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.414+0000 7ff164d99640 4 set_mon_vals no callback set 2024-03-25T17:37:08.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.718+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff15c003980 con 0x7ff1680b4d60 2024-03-25T17:37:08.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.718+0000 7ff156ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:08.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.718+0000 7ff156ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:08.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.718+0000 7ff156ffd640 10 monclient: dump: 2024-03-25T17:37:08.719 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:08.719 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:08.719 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.720 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:08.720 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:08.720 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:08.720 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:08.720 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:08.720 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:08.720 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.719+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7ff15c003b60 con 0x7ff1680b4d60 2024-03-25T17:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.720+0000 7ff16659c640 10 monclient: get_auth_request con 0x7ff14806e060 auth_method 0 2024-03-25T17:37:08.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.720+0000 7ff156ffd640 10 client.4551.objecter ms_handle_connect 0x7ff14806e060 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7ff15c053690 con 0x7ff1680b4d60 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 10 client.4551.objecter ms_dispatch 0x7ff168000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 3 client.4551.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 3 client.4551.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff15c04f1a0 con 0x7ff1680b4d60 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:08.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.726+0000 7ff156ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:08.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.943+0000 7ff16cb2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:08.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.943+0000 7ff16cb2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:08.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.943+0000 7ff16cb2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:08.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.943+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1681db110 con 0x7ff1680b4d60 2024-03-25T17:37:08.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.998+0000 7ff156ffd640 1 -- 172.21.15.134:0/77212163 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7ff1681db110 con 0x7ff1680b4d60 2024-03-25T17:37:08.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.998+0000 7ff156ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:08.999 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:08.998+0000 7ff156ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:09.000 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:09.000 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:09.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 10 librados: watch_flush enter 2024-03-25T17:37:09.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 10 librados: watch_flush exit 2024-03-25T17:37:09.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-25T17:37:09.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 10 client.4551.objecter successfully canceled tick 2024-03-25T17:37:09.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 >> v1:172.21.15.134:6800/610837766 conn(0x7ff14806e060 legacy=0x7ff148070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:09.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 10 monclient: shutdown 2024-03-25T17:37:09.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:09.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 >> v1:172.21.15.134:6789/0 conn(0x7ff1680b4d60 legacy=0x7ff1681db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:09.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 shutdown_connections 2024-03-25T17:37:09.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 >> 172.21.15.134:0/77212163 conn(0x7ff1680617a0 msgr2=0x7ff168145130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:09.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 shutdown_connections 2024-03-25T17:37:09.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 1 -- 172.21.15.134:0/77212163 wait complete. 2024-03-25T17:37:09.003 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:09.001+0000 7ff16cb2b640 1 librados: shutdown 2024-03-25T17:37:09.013 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:10.402 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:15.418 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.405 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:15.421 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.444 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:15.460 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.445 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:15.474 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.459 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:15.488 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.474 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:15.490 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.485 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:15.501 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.490 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:15.518 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:10.509 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:15.525 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:15.820 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.935+0000 7f9b1c806640 10 monclient: get_monmap_and_config 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.935+0000 7f9b1c806640 10 monclient: build_initial_monmap 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.935+0000 7f9b1c806640 10 monclient: monmap: 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:15.936453+0000 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:15.936453+0000 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:15.936 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:15.937 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:15.937 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:15.937 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:15.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 1 Processor -- start 2024-03-25T17:37:15.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 1 -- start start 2024-03-25T17:37:15.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient: init 2024-03-25T17:37:15.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:15.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:37:15.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient(hunting): picked mon.noname-a con 0x7f9b180b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:15.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient(hunting): picked mon.noname-b con 0x7f9b180b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:15.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient(hunting): picked mon.noname-c con 0x7f9b1814be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:15.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b180af940 con 0x7f9b180b4d60 2024-03-25T17:37:15.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b180b0a70 con 0x7f9b1814be90 2024-03-25T17:37:15.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b180b0d50 con 0x7f9b180b1730 2024-03-25T17:37:15.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:15.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.936+0000 7f9b1c806640 10 monclient(hunting): authenticate will time out at 5789.199586s 2024-03-25T17:37:15.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.945+0000 7f9b1659c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f9b180b4d60 0x7f9b1813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:41106/0 (socket says 172.21.15.134:41106) 2024-03-25T17:37:15.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.945+0000 7f9b1659c640 1 -- 172.21.15.134:0/2601330846 learned_addr learned my addr 172.21.15.134:0/2601330846 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:15.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.957+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350085180 0 0) 0x7f9b180b0a70 con 0x7f9b1814be90 2024-03-25T17:37:15.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.957+0000 7f9b15d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:37:15.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.957+0000 7f9b15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:15.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.957+0000 7f9b15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:15.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.957+0000 7f9b15d9b640 10 monclient(hunting): my global_id is 4585 2024-03-25T17:37:15.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.957+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b04003520 con 0x7f9b1814be90 2024-03-25T17:37:15.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.960+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639989987 0 0) 0x7f9b180af940 con 0x7f9b180b4d60 2024-03-25T17:37:15.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.960+0000 7f9b15d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:37:15.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.960+0000 7f9b15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:15.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.960+0000 7f9b15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:15.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.960+0000 7f9b15d9b640 10 monclient(hunting): my global_id is 4554 2024-03-25T17:37:15.961 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.960+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b180b0a70 con 0x7f9b180b4d60 2024-03-25T17:37:15.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 607058482 0 0) 0x7f9b180b0a70 con 0x7f9b180b4d60 2024-03-25T17:37:15.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 >> v1:172.21.15.134:6790/0 conn(0x7f9b180b1730 legacy=0x7f9b180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:15.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 >> v1:172.21.15.159:6789/0 conn(0x7f9b1814be90 legacy=0x7f9b1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:15.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:37:15.969 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 1 monclient: found mon.noname-a 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b180b1030 con 0x7f9b180b4d60 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 10 monclient: _finish_auth 0 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9b08001670 con 0x7f9b180b4d60 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b15d9b640 10 monclient: dump: 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:15.970 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b1c806640 5 monclient: authenticate success, global_id 4554 2024-03-25T17:37:15.971 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.968+0000 7f9b1c806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:15.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.980+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b080017d0 con 0x7f9b180b4d60 2024-03-25T17:37:15.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b15d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:15.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b15d9b640 1 -- 172.21.15.134:0/2601330846 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9b08001ba0 con 0x7f9b180b4d60 2024-03-25T17:37:15.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:15.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b15d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b15d9b640 10 monclient: dump: 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b1c806640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:15.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b1c806640 4 set_mon_vals no callback set 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b1c806640 10 monclient: shutdown 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b1c806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b1c806640 1 -- 172.21.15.134:0/2601330846 >> v1:172.21.15.134:6789/0 conn(0x7f9b180b4d60 legacy=0x7f9b1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b1c806640 1 -- 172.21.15.134:0/2601330846 shutdown_connections 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.981+0000 7f9b1c806640 1 -- 172.21.15.134:0/2601330846 >> 172.21.15.134:0/2601330846 conn(0x7f9b180617a0 msgr2=0x7f9b18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.982+0000 7f9b1c806640 1 -- 172.21.15.134:0/2601330846 shutdown_connections 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.982+0000 7f9b1c806640 1 -- 172.21.15.134:0/2601330846 wait complete. 2024-03-25T17:37:15.984 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.982+0000 7f9b1c806640 10 monclient: build_initial_monmap 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.982+0000 7f9b1c806640 10 monclient: monmap: 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.983+0000 7f9b1c806640 1 librados: starting msgr at 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.983+0000 7f9b1c806640 1 librados: starting objecter 2024-03-25T17:37:15.985 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.983+0000 7f9b1c806640 1 Processor -- start 2024-03-25T17:37:15.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.983+0000 7f9b1c806640 1 -- start start 2024-03-25T17:37:15.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.983+0000 7f9b1c806640 1 librados: setting wanted keys 2024-03-25T17:37:15.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.983+0000 7f9b1c806640 1 librados: calling monclient init 2024-03-25T17:37:15.986 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.983+0000 7f9b1c806640 10 monclient: init 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 10 monclient(hunting): picked mon.noname-b con 0x7f9b180b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 10 monclient(hunting): picked mon.noname-c con 0x7f9b180b4d60 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 10 monclient(hunting): picked mon.noname-a con 0x7f9b1814be90 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b181327e0 con 0x7f9b1814be90 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b18140810 con 0x7f9b180b1730 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b18137810 con 0x7f9b180b4d60 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:15.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.984+0000 7f9b1c806640 10 monclient(hunting): authenticate will time out at 5789.247313s 2024-03-25T17:37:15.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.996+0000 7f9b0dd9b640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f9b180b4d60 0x7f9b181319c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:49736/0 (socket says 172.21.15.134:49736) 2024-03-25T17:37:15.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:15.996+0000 7f9b0dd9b640 1 -- 172.21.15.134:0/1585854628 learned_addr learned my addr 172.21.15.134:0/1585854628 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:16.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.001+0000 7f9b0f7fe640 10 client.?.objecter ms_handle_connect 0x7f9b180b4d60 2024-03-25T17:37:16.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.001+0000 7f9b0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:16.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.001+0000 7f9b0f7fe640 10 client.?.objecter ms_handle_connect 0x7f9b180b1730 2024-03-25T17:37:16.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.001+0000 7f9b0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:16.007 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.007+0000 7f9b0f7fe640 10 client.?.objecter ms_handle_connect 0x7f9b1814be90 2024-03-25T17:37:16.008 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.007+0000 7f9b0f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.010+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054330994 0 0) 0x7f9b18140810 con 0x7f9b180b1730 2024-03-25T17:37:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.010+0000 7f9b0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.010+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.010+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.010+0000 7f9b0f7fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T17:37:16.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.010+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9af8004eb0 con 0x7f9b180b1730 2024-03-25T17:37:16.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967958140 0 0) 0x7f9b18137810 con 0x7f9b180b4d60 2024-03-25T17:37:16.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:16.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): my global_id is 34544 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b18140810 con 0x7f9b180b4d60 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431797918 0 0) 0x7f9b181327e0 con 0x7f9b1814be90 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 10 monclient(hunting): my global_id is 4557 2024-03-25T17:37:16.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.018+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b18137810 con 0x7f9b1814be90 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.020+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 70398869 0 0) 0x7f9af8004eb0 con 0x7f9b180b1730 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.020+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b181327e0 con 0x7f9b180b1730 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.020+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9b08001ae0 con 0x7f9b180b1730 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.020+0000 7f9b0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.020+0000 7f9b0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.020+0000 7f9b0f7fe640 10 monclient(hunting): dump: 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:16.021 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:16.022 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:16.022 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:16.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.020+0000 7f9b0f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2348740420 0 0) 0x7f9b181327e0 con 0x7f9b180b1730 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 >> v1:172.21.15.134:6790/0 conn(0x7f9b180b4d60 legacy=0x7f9b181319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 >> v1:172.21.15.134:6789/0 conn(0x7f9b1814be90 legacy=0x7f9b181320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 1 monclient: found mon.b 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:16.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b181379a0 con 0x7f9b180b1730 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 5 monclient: authenticate success, global_id 4588 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 10 monclient: _renew_subs 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b18140380 con 0x7f9b180b1730 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 10 monclient: _renew_subs 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b18138290 con 0x7f9b180b1730 2024-03-25T17:37:16.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.031+0000 7f9b1c806640 1 librados: init done 2024-03-25T17:37:16.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.032+0000 7f9b1c806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:16.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.032+0000 7f9b1c806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:16.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.032+0000 7f9b1c806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:16.034 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.032+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b180627b0 con 0x7f9b180b1730 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.311+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b08004a90 con 0x7f9b180b1730 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.311+0000 7f9b0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.311+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9b08004da0 con 0x7f9b180b1730 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.311+0000 7f9b0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.311+0000 7f9b0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.311+0000 7f9b0f7fe640 10 monclient: dump: 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:16.312 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:16.313 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:16.313 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:16.313 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:16.313 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:16.313 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:16.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.311+0000 7f9b1559a640 4 set_mon_vals no callback set 2024-03-25T17:37:16.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.312+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f9b0801ad70 con 0x7f9b180b1730 2024-03-25T17:37:16.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.312+0000 7f9b0dd9b640 10 monclient: get_auth_request con 0x7f9af806dda0 auth_method 0 2024-03-25T17:37:16.314 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.313+0000 7f9b0f7fe640 10 client.4588.objecter ms_handle_connect 0x7f9af806dda0 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.317+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f9b080548f0 con 0x7f9b180b1730 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.317+0000 7f9b0f7fe640 10 client.4588.objecter ms_dispatch 0x7f9b18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.317+0000 7f9b0f7fe640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.317+0000 7f9b0f7fe640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.317+0000 7f9b0f7fe640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.317+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b08050230 con 0x7f9b180b1730 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.318+0000 7f9b0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:16.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.318+0000 7f9b0f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:16.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.525+0000 7f9b1c806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:16.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.525+0000 7f9b1c806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:16.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.525+0000 7f9b1c806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:37:16.526 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.525+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b18140d60 con 0x7f9b180b1730 2024-03-25T17:37:16.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.589+0000 7f9b0f7fe640 1 -- 172.21.15.134:0/1585854628 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f9b08052330 con 0x7f9b180b1730 2024-03-25T17:37:16.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.589+0000 7f9b0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:16.590 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.589+0000 7f9b0f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:16.590 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:16.590 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:16.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 10 librados: watch_flush enter 2024-03-25T17:37:16.592 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 10 librados: watch_flush exit 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 10 client.4588.objecter successfully canceled tick 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 >> v1:172.21.15.134:6800/610837766 conn(0x7f9af806dda0 legacy=0x7f9af8070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 10 monclient: shutdown 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 >> v1:172.21.15.159:6789/0 conn(0x7f9b180b1730 legacy=0x7f9b181d7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 shutdown_connections 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 >> 172.21.15.134:0/1585854628 conn(0x7f9b180617a0 msgr2=0x7f9b180b6b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:16.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 shutdown_connections 2024-03-25T17:37:16.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 1 -- 172.21.15.134:0/1585854628 wait complete. 2024-03-25T17:37:16.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:16.591+0000 7f9b1c806640 1 librados: shutdown 2024-03-25T17:37:16.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:23.410 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.528+0000 7f7d6b825640 10 monclient: get_monmap_and_config 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.528+0000 7f7d6b825640 10 monclient: build_initial_monmap 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.528+0000 7f7d6b825640 10 monclient: monmap: 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:23.530574+0000 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:23.530574+0000 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:23.530 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:23.531 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:23.531 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:23.531 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:23.531 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:23.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.529+0000 7f7d6b825640 1 Processor -- start 2024-03-25T17:37:23.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.529+0000 7f7d6b825640 1 -- start start 2024-03-25T17:37:23.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.529+0000 7f7d6b825640 10 monclient: init 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 10 monclient(hunting): picked mon.noname-c con 0x7f7d640a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 10 monclient(hunting): picked mon.noname-a con 0x7f7d6413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 10 monclient(hunting): picked mon.noname-b con 0x7f7d641430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d64142520 con 0x7f7d6413f710 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640b28a0 con 0x7f7d640a98f0 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d640b2a30 con 0x7f7d641430c0 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:23.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.530+0000 7f7d6b825640 10 monclient(hunting): authenticate will time out at 5796.794342s 2024-03-25T17:37:23.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.538+0000 7f7d69dc2640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f7d641430c0 0x7f7d6414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33032/0 (socket says 172.21.15.134:33032) 2024-03-25T17:37:23.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.538+0000 7f7d69dc2640 1 -- 172.21.15.134:0/289319747 learned_addr learned my addr 172.21.15.134:0/289319747 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:23.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.552+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342420835 0 0) 0x7f7d640b2a30 con 0x7f7d641430c0 2024-03-25T17:37:23.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.552+0000 7f7d5bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:23.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.552+0000 7f7d5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:23.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.552+0000 7f7d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:23.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.552+0000 7f7d5bfff640 10 monclient(hunting): my global_id is 34547 2024-03-25T17:37:23.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.552+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d50003520 con 0x7f7d641430c0 2024-03-25T17:37:23.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.553+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275430898 0 0) 0x7f7d64142520 con 0x7f7d6413f710 2024-03-25T17:37:23.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.553+0000 7f7d5bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:23.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.553+0000 7f7d5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:23.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.553+0000 7f7d5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:23.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.553+0000 7f7d5bfff640 10 monclient(hunting): my global_id is 4560 2024-03-25T17:37:23.555 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.553+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d640b2a30 con 0x7f7d6413f710 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1590662057 0 0) 0x7f7d50003520 con 0x7f7d641430c0 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 >> v1:172.21.15.159:6789/0 conn(0x7f7d640a98f0 legacy=0x7f7d640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 >> v1:172.21.15.134:6789/0 conn(0x7f7d6413f710 legacy=0x7f7d641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 1 monclient: found mon.noname-b 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d640b44c0 con 0x7f7d641430c0 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 10 monclient: _finish_auth 0 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:37:23.569 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7d5c001a10 con 0x7f7d641430c0 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d6b825640 5 monclient: authenticate success, global_id 34547 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d6b825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d5bfff640 10 monclient: dump: 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.570 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:23.571 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:23.571 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:23.571 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:23.571 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:23.571 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:23.571 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.567+0000 7f7d6b825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:23.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d5c001b70 con 0x7f7d641430c0 2024-03-25T17:37:23.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:23.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d5bfff640 1 -- 172.21.15.134:0/289319747 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7d5c001f80 con 0x7f7d641430c0 2024-03-25T17:37:23.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:23.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d5bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:23.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d5bfff640 10 monclient: dump: 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d6b825640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:23.595 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.592+0000 7f7d6b825640 4 set_mon_vals no callback set 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.593+0000 7f7d6b825640 10 monclient: shutdown 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.593+0000 7f7d6b825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.593+0000 7f7d6b825640 1 -- 172.21.15.134:0/289319747 >> v1:172.21.15.134:6790/0 conn(0x7f7d641430c0 legacy=0x7f7d6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.593+0000 7f7d6b825640 1 -- 172.21.15.134:0/289319747 shutdown_connections 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.593+0000 7f7d6b825640 1 -- 172.21.15.134:0/289319747 >> 172.21.15.134:0/289319747 conn(0x7f7d640b7ea0 msgr2=0x7f7d640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.593+0000 7f7d6b825640 1 -- 172.21.15.134:0/289319747 shutdown_connections 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.593+0000 7f7d6b825640 1 -- 172.21.15.134:0/289319747 wait complete. 2024-03-25T17:37:23.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.594+0000 7f7d6b825640 10 monclient: build_initial_monmap 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.594+0000 7f7d6b825640 10 monclient: monmap: 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:23.599 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.595+0000 7f7d6b825640 1 librados: starting msgr at 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.595+0000 7f7d6b825640 1 librados: starting objecter 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.595+0000 7f7d6b825640 1 Processor -- start 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.595+0000 7f7d6b825640 1 -- start start 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.595+0000 7f7d6b825640 1 librados: setting wanted keys 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.595+0000 7f7d6b825640 1 librados: calling monclient init 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.595+0000 7f7d6b825640 10 monclient: init 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 10 monclient(hunting): picked mon.noname-c con 0x7f7d640a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 10 monclient(hunting): picked mon.noname-b con 0x7f7d6413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:23.600 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 10 monclient(hunting): picked mon.noname-a con 0x7f7d641430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:23.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6413c9b0 con 0x7f7d641430c0 2024-03-25T17:37:23.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d641397c0 con 0x7f7d6413f710 2024-03-25T17:37:23.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d64139950 con 0x7f7d640a98f0 2024-03-25T17:37:23.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:23.601 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.596+0000 7f7d6b825640 10 monclient(hunting): authenticate will time out at 5796.860245s 2024-03-25T17:37:23.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.607+0000 7f7d695c1640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f7d640a98f0 0x7f7d64130f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:33086/0 (socket says 172.21.15.134:33086) 2024-03-25T17:37:23.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.607+0000 7f7d68dc0640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f7d6413f710 0x7f7d641dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:56174/0 (socket says 172.21.15.134:56174) 2024-03-25T17:37:23.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.607+0000 7f7d695c1640 1 -- 172.21.15.134:0/3970423678 learned_addr learned my addr 172.21.15.134:0/3970423678 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:23.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.612+0000 7f7d59ffb640 10 client.?.objecter ms_handle_connect 0x7f7d6413f710 2024-03-25T17:37:23.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.612+0000 7f7d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:23.615 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.614+0000 7f7d59ffb640 10 client.?.objecter ms_handle_connect 0x7f7d640a98f0 2024-03-25T17:37:23.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.614+0000 7f7d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:23.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.614+0000 7f7d59ffb640 10 client.?.objecter ms_handle_connect 0x7f7d641430c0 2024-03-25T17:37:23.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.614+0000 7f7d59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:23.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.618+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803337432 0 0) 0x7f7d6413c9b0 con 0x7f7d641430c0 2024-03-25T17:37:23.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.619+0000 7f7d59ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:23.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.619+0000 7f7d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:23.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.619+0000 7f7d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:23.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.619+0000 7f7d59ffb640 10 monclient(hunting): my global_id is 4563 2024-03-25T17:37:23.620 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.619+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d48004eb0 con 0x7f7d641430c0 2024-03-25T17:37:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.622+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060692991 0 0) 0x7f7d641397c0 con 0x7f7d6413f710 2024-03-25T17:37:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.622+0000 7f7d59ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.622+0000 7f7d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.622+0000 7f7d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.622+0000 7f7d59ffb640 10 monclient(hunting): my global_id is 4594 2024-03-25T17:37:23.624 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.622+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6413c9b0 con 0x7f7d6413f710 2024-03-25T17:37:23.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.626+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420876080 0 0) 0x7f7d64139950 con 0x7f7d640a98f0 2024-03-25T17:37:23.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.626+0000 7f7d59ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:23.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.626+0000 7f7d59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:23.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.626+0000 7f7d59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:23.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.626+0000 7f7d59ffb640 10 monclient(hunting): my global_id is 34550 2024-03-25T17:37:23.628 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.626+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d641397c0 con 0x7f7d640a98f0 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.629+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 290333475 0 0) 0x7f7d48004eb0 con 0x7f7d641430c0 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.629+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d64139950 con 0x7f7d641430c0 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.629+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7d5c001e80 con 0x7f7d641430c0 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.629+0000 7f7d59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.629+0000 7f7d59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.629+0000 7f7d59ffb640 10 monclient(hunting): dump: 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.631 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:23.632 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:23.632 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:23.632 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:23.632 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:23.632 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:23.632 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.629+0000 7f7d59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1184342401 0 0) 0x7f7d64139950 con 0x7f7d641430c0 2024-03-25T17:37:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 >> v1:172.21.15.134:6790/0 conn(0x7f7d640a98f0 legacy=0x7f7d64130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 >> v1:172.21.15.159:6789/0 conn(0x7f7d6413f710 legacy=0x7f7d641dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:37:23.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 1 monclient: found mon.a 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64131680 con 0x7f7d641430c0 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 10 monclient: _finish_auth 0 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 5 monclient: authenticate success, global_id 4563 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 10 monclient: _renew_subs 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d64131950 con 0x7f7d641430c0 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:23.641 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 10 monclient: _renew_subs 2024-03-25T17:37:23.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:23.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d64135a30 con 0x7f7d641430c0 2024-03-25T17:37:23.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.638+0000 7f7d6b825640 1 librados: init done 2024-03-25T17:37:23.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.639+0000 7f7d6b825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:23.642 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.639+0000 7f7d6b825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:23.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.639+0000 7f7d6b825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:23.643 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.639+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6413c810 con 0x7f7d641430c0 2024-03-25T17:37:23.934 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.932+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7d5c002040 con 0x7f7d641430c0 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.932+0000 7f7d59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.932+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7d5c004e10 con 0x7f7d641430c0 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.932+0000 7f7d59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.932+0000 7f7d59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.932+0000 7f7d59ffb640 10 monclient: dump: 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:23.935 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:23.936 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:23.936 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:23.936 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:23.936 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:23.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.932+0000 7f7d5b7fe640 4 set_mon_vals no callback set 2024-03-25T17:37:23.936 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.933+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f7d5c01ade0 con 0x7f7d641430c0 2024-03-25T17:37:23.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.934+0000 7f7d695c1640 10 monclient: get_auth_request con 0x7f7d4806ddf0 auth_method 0 2024-03-25T17:37:23.937 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.934+0000 7f7d59ffb640 10 client.4563.objecter ms_handle_connect 0x7f7d4806ddf0 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f7d5c0548b0 con 0x7f7d641430c0 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 10 client.4563.objecter ms_dispatch 0x7f7d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 3 client.4563.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 3 client.4563.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7d5c0503c0 con 0x7f7d641430c0 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:23.943 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:23.941+0000 7f7d59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:24.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.160+0000 7f7d6b825640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:24.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.160+0000 7f7d6b825640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:24.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.160+0000 7f7d6b825640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:24.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.160+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d64136fc0 con 0x7f7d641430c0 2024-03-25T17:37:24.207 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.206+0000 7f7d59ffb640 1 -- 172.21.15.134:0/3970423678 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f7d5c0522f0 con 0x7f7d641430c0 2024-03-25T17:37:24.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.206+0000 7f7d59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:24.208 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.206+0000 7f7d59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:24.208 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:24.208 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.208+0000 7f7d6b825640 10 librados: watch_flush enter 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.208+0000 7f7d6b825640 10 librados: watch_flush exit 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.208+0000 7f7d6b825640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.208+0000 7f7d6b825640 10 client.4563.objecter successfully canceled tick 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 >> v1:172.21.15.134:6800/610837766 conn(0x7f7d4806ddf0 legacy=0x7f7d480702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 10 monclient: shutdown 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 >> v1:172.21.15.134:6789/0 conn(0x7f7d641430c0 legacy=0x7f7d641e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:24.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 shutdown_connections 2024-03-25T17:37:24.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 >> 172.21.15.134:0/3970423678 conn(0x7f7d640b7ea0 msgr2=0x7f7d6414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:24.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 shutdown_connections 2024-03-25T17:37:24.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 1 -- 172.21.15.134:0/3970423678 wait complete. 2024-03-25T17:37:24.212 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:24.209+0000 7f7d6b825640 1 librados: shutdown 2024-03-25T17:37:24.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:30.480 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:35.497 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.492 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:35.508 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.520 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:35.536 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.541 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:35.570 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.548 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:35.577 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.553 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:35.570 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.572 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:35.588 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.574 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:35.603 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:30.606 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:35.622 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:31.029 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.143+0000 7fb57d501640 10 monclient: get_monmap_and_config 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.143+0000 7fb57d501640 10 monclient: build_initial_monmap 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.143+0000 7fb57d501640 10 monclient: monmap: 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:31.144482+0000 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:31.144482+0000 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:31.144 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:31.145 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:31.145 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:31.145 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 1 Processor -- start 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 1 -- start start 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient: init 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient(hunting): picked mon.noname-b con 0x7fb5780705b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient(hunting): picked mon.noname-a con 0x7fb5781402d0 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient(hunting): picked mon.noname-c con 0x7fb578149ae0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5780718c0 con 0x7fb5781402d0 2024-03-25T17:37:31.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5780714c0 con 0x7fb578149ae0 2024-03-25T17:37:31.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb578072b50 con 0x7fb5780705b0 2024-03-25T17:37:31.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:31.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.144+0000 7fb57d501640 10 monclient(hunting): authenticate will time out at 5804.407575s 2024-03-25T17:37:31.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.151+0000 7fb5777fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb578149ae0 0x7fb57814bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:39958/0 (socket says 172.21.15.134:39958) 2024-03-25T17:37:31.151 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.151+0000 7fb5777fe640 1 -- 172.21.15.134:0/3543253106 learned_addr learned my addr 172.21.15.134:0/3543253106 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:31.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.166+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090094371 0 0) 0x7fb578072b50 con 0x7fb5780705b0 2024-03-25T17:37:31.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.166+0000 7fb575ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:31.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.166+0000 7fb575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:31.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.166+0000 7fb575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:31.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.166+0000 7fb575ffb640 10 monclient(hunting): my global_id is 34553 2024-03-25T17:37:31.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.166+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb560003520 con 0x7fb5780705b0 2024-03-25T17:37:31.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.170+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962679877 0 0) 0x7fb5780714c0 con 0x7fb578149ae0 2024-03-25T17:37:31.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.170+0000 7fb575ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:31.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.170+0000 7fb575ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:31.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.170+0000 7fb575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:31.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.170+0000 7fb575ffb640 10 monclient(hunting): my global_id is 4597 2024-03-25T17:37:31.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.170+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb578072b50 con 0x7fb578149ae0 2024-03-25T17:37:31.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3115055940 0 0) 0x7fb578072b50 con 0x7fb578149ae0 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 >> v1:172.21.15.134:6790/0 conn(0x7fb5780705b0 legacy=0x7fb57813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 >> v1:172.21.15.134:6789/0 conn(0x7fb5781402d0 legacy=0x7fb578140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 1 monclient: found mon.noname-c 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57813dd10 con 0x7fb578149ae0 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 10 monclient: _finish_auth 0 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:31.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb5700019f0 con 0x7fb578149ae0 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb575ffb640 10 monclient: dump: 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:31.187 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:31.188 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:31.188 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:31.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb57d501640 5 monclient: authenticate success, global_id 4597 2024-03-25T17:37:31.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.184+0000 7fb57d501640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb570001b50 con 0x7fb578149ae0 2024-03-25T17:37:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb575ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb575ffb640 1 -- 172.21.15.134:0/3543253106 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb570001f60 con 0x7fb578149ae0 2024-03-25T17:37:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb57d501640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:31.200 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb57d501640 4 set_mon_vals no callback set 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb575ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb575ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.195+0000 7fb575ffb640 10 monclient: dump: 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:31.201 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.196+0000 7fb57d501640 10 monclient: shutdown 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.196+0000 7fb57d501640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.196+0000 7fb57d501640 1 -- 172.21.15.134:0/3543253106 >> v1:172.21.15.159:6789/0 conn(0x7fb578149ae0 legacy=0x7fb57814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.196+0000 7fb57d501640 1 -- 172.21.15.134:0/3543253106 shutdown_connections 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.196+0000 7fb57d501640 1 -- 172.21.15.134:0/3543253106 >> 172.21.15.134:0/3543253106 conn(0x7fb5780b5d10 msgr2=0x7fb5780b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.196+0000 7fb57d501640 1 -- 172.21.15.134:0/3543253106 shutdown_connections 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.196+0000 7fb57d501640 1 -- 172.21.15.134:0/3543253106 wait complete. 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 10 monclient: build_initial_monmap 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 10 monclient: monmap: 2024-03-25T17:37:31.202 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 1 librados: starting msgr at 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 1 librados: starting objecter 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 1 Processor -- start 2024-03-25T17:37:31.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 1 -- start start 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 1 librados: setting wanted keys 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 1 librados: calling monclient init 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.197+0000 7fb57d501640 10 monclient: init 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 10 monclient(hunting): picked mon.noname-c con 0x7fb5780705b0 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 10 monclient(hunting): picked mon.noname-b con 0x7fb5781402d0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 10 monclient(hunting): picked mon.noname-a con 0x7fb578149ae0 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57806d360 con 0x7fb578149ae0 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57813f8b0 con 0x7fb5781402d0 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb578136f20 con 0x7fb5780705b0 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:31.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.198+0000 7fb57d501640 10 monclient(hunting): authenticate will time out at 5804.461468s 2024-03-25T17:37:31.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.209+0000 7fb5767fc640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fb5781402d0 0x7fb578133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:39984/0 (socket says 172.21.15.134:39984) 2024-03-25T17:37:31.210 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.209+0000 7fb5767fc640 1 -- 172.21.15.134:0/3769394161 learned_addr learned my addr 172.21.15.134:0/3769394161 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:31.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.215+0000 7fb55f7fe640 10 client.?.objecter ms_handle_connect 0x7fb5781402d0 2024-03-25T17:37:31.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.215+0000 7fb55f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:31.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.217+0000 7fb55f7fe640 10 client.?.objecter ms_handle_connect 0x7fb5780705b0 2024-03-25T17:37:31.218 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.217+0000 7fb55f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:31.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 10 client.?.objecter ms_handle_connect 0x7fb578149ae0 2024-03-25T17:37:31.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:31.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514220842 0 0) 0x7fb57813f8b0 con 0x7fb5781402d0 2024-03-25T17:37:31.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:31.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:31.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:31.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 10 monclient(hunting): my global_id is 4600 2024-03-25T17:37:31.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.228+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb564004eb0 con 0x7fb5781402d0 2024-03-25T17:37:31.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.231+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398128335 0 0) 0x7fb578136f20 con 0x7fb5780705b0 2024-03-25T17:37:31.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.231+0000 7fb55f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:31.231 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.231+0000 7fb55f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:31.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.231+0000 7fb55f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:31.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.231+0000 7fb55f7fe640 10 monclient(hunting): my global_id is 34556 2024-03-25T17:37:31.232 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.231+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb57813f8b0 con 0x7fb5780705b0 2024-03-25T17:37:31.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.242+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087605625 0 0) 0x7fb57806d360 con 0x7fb578149ae0 2024-03-25T17:37:31.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.242+0000 7fb55f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:31.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.242+0000 7fb55f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:31.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.242+0000 7fb55f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:31.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.242+0000 7fb55f7fe640 10 monclient(hunting): my global_id is 4569 2024-03-25T17:37:31.243 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.242+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb578136f20 con 0x7fb578149ae0 2024-03-25T17:37:31.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.244+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2557291952 0 0) 0x7fb57813f8b0 con 0x7fb5780705b0 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.244+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb57806d360 con 0x7fb5780705b0 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.244+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb568001950 con 0x7fb5780705b0 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.244+0000 7fb55f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.244+0000 7fb55f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.244+0000 7fb55f7fe640 10 monclient(hunting): dump: 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:31.245 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:31.246 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:31.246 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:31.246 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.244+0000 7fb55f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:31.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.251+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3483126086 0 0) 0x7fb564004eb0 con 0x7fb5781402d0 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.251+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb57813f8b0 con 0x7fb5781402d0 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.251+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb56c001a30 con 0x7fb5781402d0 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.251+0000 7fb55f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.251+0000 7fb55f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.251+0000 7fb55f7fe640 10 monclient(hunting): dump: 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:31.252 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:31.253 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:31.253 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:31.253 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:31.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 994131633 0 0) 0x7fb57806d360 con 0x7fb5780705b0 2024-03-25T17:37:31.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 >> v1:172.21.15.159:6789/0 conn(0x7fb5781402d0 legacy=0x7fb578133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:31.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 >> v1:172.21.15.134:6789/0 conn(0x7fb578149ae0 legacy=0x7fb5781e2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:31.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:37:31.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 1 monclient: found mon.c 2024-03-25T17:37:31.255 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57813fdd0 con 0x7fb5780705b0 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb55f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 5 monclient: authenticate success, global_id 34556 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 10 monclient: _renew_subs 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb578137850 con 0x7fb5780705b0 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 10 client.34556.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 10 monclient: _renew_subs 2024-03-25T17:37:31.256 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:31.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5781d7bf0 con 0x7fb5780705b0 2024-03-25T17:37:31.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.253+0000 7fb57d501640 1 librados: init done 2024-03-25T17:37:31.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.254+0000 7fb57d501640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:31.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.254+0000 7fb57d501640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:31.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.254+0000 7fb57d501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:31.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.254+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5781370b0 con 0x7fb5780705b0 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.551+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb568001d60 con 0x7fb5780705b0 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.551+0000 7fb55f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.551+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fb5680037b0 con 0x7fb5780705b0 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.551+0000 7fb55f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.551+0000 7fb55f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.551+0000 7fb55f7fe640 10 monclient: dump: 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:31.552 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:31.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.551+0000 7fb5757fa640 4 set_mon_vals no callback set 2024-03-25T17:37:31.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.552+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fb5680039d0 con 0x7fb5780705b0 2024-03-25T17:37:31.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.553+0000 7fb5767fc640 10 monclient: get_auth_request con 0x7fb56406df70 auth_method 0 2024-03-25T17:37:31.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.553+0000 7fb55f7fe640 10 client.34556.objecter ms_handle_connect 0x7fb56406df70 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fb5680534d0 con 0x7fb5780705b0 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 10 client.34556.objecter ms_dispatch 0x7fb578000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 3 client.34556.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 3 client.34556.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 20 client.34556.objecter dump_active .. 0 homeless 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb56804efe0 con 0x7fb5780705b0 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:31.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.558+0000 7fb55f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:31.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.780+0000 7fb57d501640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:31.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.780+0000 7fb57d501640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:31.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.780+0000 7fb57d501640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:31.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.780+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5780627b0 con 0x7fb5780705b0 2024-03-25T17:37:31.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.818+0000 7fb55f7fe640 1 -- 172.21.15.134:0/3769394161 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fb568050f10 con 0x7fb5780705b0 2024-03-25T17:37:31.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.818+0000 7fb55f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:31.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.818+0000 7fb55f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:31.819 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:31.820 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:31.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 10 librados: watch_flush enter 2024-03-25T17:37:31.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 10 librados: watch_flush exit 2024-03-25T17:37:31.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 20 client.34556.objecter shutdown clearing up homeless session... 2024-03-25T17:37:31.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 10 client.34556.objecter successfully canceled tick 2024-03-25T17:37:31.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 >> v1:172.21.15.134:6800/610837766 conn(0x7fb56406df70 legacy=0x7fb564070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:31.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 10 monclient: shutdown 2024-03-25T17:37:31.822 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:31.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 >> v1:172.21.15.134:6790/0 conn(0x7fb5780705b0 legacy=0x7fb5781d7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:31.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 shutdown_connections 2024-03-25T17:37:31.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 >> 172.21.15.134:0/3769394161 conn(0x7fb5780b5d10 msgr2=0x7fb578140710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:31.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 shutdown_connections 2024-03-25T17:37:31.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 1 -- 172.21.15.134:0/3769394161 wait complete. 2024-03-25T17:37:31.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:31.820+0000 7fb57d501640 1 librados: shutdown 2024-03-25T17:37:31.834 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:38.641 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.757+0000 7fe9cdac9640 10 monclient: get_monmap_and_config 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.757+0000 7fe9cdac9640 10 monclient: build_initial_monmap 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.757+0000 7fe9cdac9640 10 monclient: monmap: 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:38.758483+0000 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:38.758483+0000 2024-03-25T17:37:38.758 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:38.759 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:38.759 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:38.759 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:38.759 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:38.759 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:38.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.758+0000 7fe9cdac9640 1 Processor -- start 2024-03-25T17:37:38.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.758+0000 7fe9cdac9640 1 -- start start 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.758+0000 7fe9cdac9640 10 monclient: init 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c8142520 con 0x7fe9c80a98f0 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c80b28a0 con 0x7fe9c813f710 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c80b2a30 con 0x7fe9c81430c0 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:38.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.759+0000 7fe9cdac9640 10 monclient(hunting): authenticate will time out at 5812.021937s 2024-03-25T17:37:38.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.768+0000 7fe9c6ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe9c813f710 0x7fe9c81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:44648/0 (socket says 172.21.15.134:44648) 2024-03-25T17:37:38.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.768+0000 7fe9c6ffd640 1 -- 172.21.15.134:0/4251808070 learned_addr learned my addr 172.21.15.134:0/4251808070 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:38.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.782+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888171039 0 0) 0x7fe9c80b28a0 con 0x7fe9c813f710 2024-03-25T17:37:38.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.782+0000 7fe9c67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:37:38.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.782+0000 7fe9c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:38.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.782+0000 7fe9c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:38.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.782+0000 7fe9c67fc640 10 monclient(hunting): my global_id is 4603 2024-03-25T17:37:38.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.782+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b0003520 con 0x7fe9c813f710 2024-03-25T17:37:38.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.790+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055085210 0 0) 0x7fe9c8142520 con 0x7fe9c80a98f0 2024-03-25T17:37:38.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.790+0000 7fe9c67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:37:38.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.790+0000 7fe9c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:38.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.790+0000 7fe9c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:38.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.790+0000 7fe9c67fc640 10 monclient(hunting): my global_id is 4572 2024-03-25T17:37:38.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.790+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c80b28a0 con 0x7fe9c80a98f0 2024-03-25T17:37:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1145977636 0 0) 0x7fe9b0003520 con 0x7fe9c813f710 2024-03-25T17:37:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 >> v1:172.21.15.134:6790/0 conn(0x7fe9c81430c0 legacy=0x7fe9c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 >> v1:172.21.15.134:6789/0 conn(0x7fe9c80a98f0 legacy=0x7fe9c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 10 monclient: _finish_hunting 0 2024-03-25T17:37:38.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 1 monclient: found mon.noname-c 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80b44c0 con 0x7fe9c813f710 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 10 monclient: _finish_auth 0 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 10 monclient: _check_auth_tickets 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe9bc001690 con 0x7fe9c813f710 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9cdac9640 5 monclient: authenticate success, global_id 4603 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9cdac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:38.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9c67fc640 10 monclient: dump: 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:38.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.798+0000 7fe9cdac9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:38.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.803+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9bc0017f0 con 0x7fe9c813f710 2024-03-25T17:37:38.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9c67fc640 1 -- 172.21.15.134:0/4251808070 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe9bc001c00 con 0x7fe9c813f710 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9c67fc640 10 monclient: dump: 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:38.805 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:38.806 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:38.806 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:38.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9cdac9640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:38.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9cdac9640 4 set_mon_vals no callback set 2024-03-25T17:37:38.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9cdac9640 10 monclient: shutdown 2024-03-25T17:37:38.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9cdac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:38.808 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.804+0000 7fe9cdac9640 1 -- 172.21.15.134:0/4251808070 >> v1:172.21.15.159:6789/0 conn(0x7fe9c813f710 legacy=0x7fe9c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.805+0000 7fe9cdac9640 1 -- 172.21.15.134:0/4251808070 shutdown_connections 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.805+0000 7fe9cdac9640 1 -- 172.21.15.134:0/4251808070 >> 172.21.15.134:0/4251808070 conn(0x7fe9c80b7ea0 msgr2=0x7fe9c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.805+0000 7fe9cdac9640 1 -- 172.21.15.134:0/4251808070 shutdown_connections 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.805+0000 7fe9cdac9640 1 -- 172.21.15.134:0/4251808070 wait complete. 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 10 monclient: build_initial_monmap 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 10 monclient: monmap: 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:38.809 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 1 librados: starting msgr at 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 1 librados: starting objecter 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 1 Processor -- start 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 1 -- start start 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 1 librados: setting wanted keys 2024-03-25T17:37:38.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 1 librados: calling monclient init 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.806+0000 7fe9cdac9640 10 monclient: init 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 10 monclient(hunting): picked mon.noname-c con 0x7fe9c80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 10 monclient(hunting): picked mon.noname-b con 0x7fe9c813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 10 monclient(hunting): picked mon.noname-a con 0x7fe9c81430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c80abdc0 con 0x7fe9c81430c0 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c81d7730 con 0x7fe9c813f710 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c8138f10 con 0x7fe9c80a98f0 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:38.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.807+0000 7fe9cdac9640 10 monclient(hunting): authenticate will time out at 5812.070381s 2024-03-25T17:37:38.818 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.818+0000 7fe9c6ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fe9c813f710 0x7fe9c81d7c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:44684/0 (socket says 172.21.15.134:44684) 2024-03-25T17:37:38.819 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.818+0000 7fe9c6ffd640 1 -- 172.21.15.134:0/1851915382 learned_addr learned my addr 172.21.15.134:0/1851915382 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:38.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.821+0000 7fe9affff640 10 client.?.objecter ms_handle_connect 0x7fe9c813f710 2024-03-25T17:37:38.821 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.821+0000 7fe9affff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:38.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.824+0000 7fe9affff640 10 client.?.objecter ms_handle_connect 0x7fe9c80a98f0 2024-03-25T17:37:38.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.824+0000 7fe9affff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:38.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.826+0000 7fe9affff640 10 client.?.objecter ms_handle_connect 0x7fe9c81430c0 2024-03-25T17:37:38.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.826+0000 7fe9affff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:38.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.838+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637131355 0 0) 0x7fe9c80abdc0 con 0x7fe9c81430c0 2024-03-25T17:37:38.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.838+0000 7fe9affff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:38.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.838+0000 7fe9affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:38.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.838+0000 7fe9affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:38.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.838+0000 7fe9affff640 10 monclient(hunting): my global_id is 4575 2024-03-25T17:37:38.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.838+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b4004eb0 con 0x7fe9c81430c0 2024-03-25T17:37:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.840+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355209638 0 0) 0x7fe9c81d7730 con 0x7fe9c813f710 2024-03-25T17:37:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.840+0000 7fe9affff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.840+0000 7fe9affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.840+0000 7fe9affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.840+0000 7fe9affff640 10 monclient(hunting): my global_id is 4606 2024-03-25T17:37:38.841 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.840+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c80abdc0 con 0x7fe9c813f710 2024-03-25T17:37:38.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.844+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280397365 0 0) 0x7fe9c8138f10 con 0x7fe9c80a98f0 2024-03-25T17:37:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.844+0000 7fe9affff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.844+0000 7fe9affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.844+0000 7fe9affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.844+0000 7fe9affff640 10 monclient(hunting): my global_id is 34562 2024-03-25T17:37:38.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.844+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c81d7730 con 0x7fe9c80a98f0 2024-03-25T17:37:38.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.851+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3806497118 0 0) 0x7fe9c81d7730 con 0x7fe9c80a98f0 2024-03-25T17:37:38.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.851+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c8138f10 con 0x7fe9c80a98f0 2024-03-25T17:37:38.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.851+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe9b8001930 con 0x7fe9c80a98f0 2024-03-25T17:37:38.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.851+0000 7fe9affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:38.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.851+0000 7fe9affff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:38.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.851+0000 7fe9affff640 10 monclient(hunting): dump: 2024-03-25T17:37:38.852 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:38.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.851+0000 7fe9affff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.858+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168712219 0 0) 0x7fe9b4004eb0 con 0x7fe9c81430c0 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.858+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9c81d7730 con 0x7fe9c81430c0 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.858+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe9c0001b20 con 0x7fe9c81430c0 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.858+0000 7fe9affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.858+0000 7fe9affff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.858+0000 7fe9affff640 10 monclient(hunting): dump: 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:38.859 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.860 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.860 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:38.860 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:38.860 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:38.860 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:38.860 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:38.860 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:38.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4247179869 0 0) 0x7fe9c80abdc0 con 0x7fe9c813f710 2024-03-25T17:37:38.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9b4004eb0 con 0x7fe9c813f710 2024-03-25T17:37:38.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe9bc001bc0 con 0x7fe9c813f710 2024-03-25T17:37:38.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:38.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:38.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 10 monclient(hunting): dump: 2024-03-25T17:37:38.867 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2833776233 0 0) 0x7fe9c81d7730 con 0x7fe9c81430c0 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 >> v1:172.21.15.134:6790/0 conn(0x7fe9c80a98f0 legacy=0x7fe9c8138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 >> v1:172.21.15.159:6789/0 conn(0x7fe9c813f710 legacy=0x7fe9c81d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:38.868 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 10 monclient: _finish_hunting 0 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 monclient: found mon.a 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c81390a0 con 0x7fe9c81430c0 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 10 monclient: _finish_auth 0 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 10 monclient: _check_auth_tickets 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9cdac9640 5 monclient: authenticate success, global_id 4575 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.865+0000 7fe9cdac9640 10 monclient: _renew_subs 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.866+0000 7fe9cdac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.866+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c81d8320 con 0x7fe9c81430c0 2024-03-25T17:37:38.869 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.866+0000 7fe9cdac9640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:38.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.866+0000 7fe9cdac9640 10 monclient: _renew_subs 2024-03-25T17:37:38.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.866+0000 7fe9cdac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:38.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.866+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c81d43c0 con 0x7fe9c81430c0 2024-03-25T17:37:38.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.866+0000 7fe9cdac9640 1 librados: init done 2024-03-25T17:37:38.870 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.867+0000 7fe9cdac9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:38.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.867+0000 7fe9cdac9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:38.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.867+0000 7fe9cdac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:38.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.867+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c80627b0 con 0x7fe9c81430c0 2024-03-25T17:37:38.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.870+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9c0002030 con 0x7fe9c81430c0 2024-03-25T17:37:38.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.870+0000 7fe9affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:38.871 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:38.870+0000 7fe9c5ffb640 4 set_mon_vals no callback set 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.169+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fe9c0003b10 con 0x7fe9c81430c0 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.170+0000 7fe9affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.170+0000 7fe9affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.170+0000 7fe9affff640 10 monclient: dump: 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:39.171 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:39.172 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:39.172 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:39.172 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:39.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.170+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fe9c0019ae0 con 0x7fe9c81430c0 2024-03-25T17:37:39.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.171+0000 7fe9c77fe640 10 monclient: get_auth_request con 0x7fe9b406e110 auth_method 0 2024-03-25T17:37:39.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.171+0000 7fe9affff640 10 client.4575.objecter ms_handle_connect 0x7fe9b406e110 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fe9c00537d0 con 0x7fe9c81430c0 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 10 client.4575.objecter ms_dispatch 0x7fe9c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 3 client.4575.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 3 client.4575.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9c004f110 con 0x7fe9c81430c0 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:39.180 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.179+0000 7fe9affff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:39.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.434+0000 7fe9cdac9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:39.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.434+0000 7fe9cdac9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:39.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.434+0000 7fe9cdac9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:39.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.434+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9c81d84b0 con 0x7fe9c81430c0 2024-03-25T17:37:39.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.458+0000 7fe9affff640 1 -- 172.21.15.134:0/1851915382 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fe9c0051210 con 0x7fe9c81430c0 2024-03-25T17:37:39.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.458+0000 7fe9affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:39.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.458+0000 7fe9affff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:39.459 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:39.459 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.460+0000 7fe9cdac9640 10 librados: watch_flush enter 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.460+0000 7fe9cdac9640 10 librados: watch_flush exit 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.460+0000 7fe9cdac9640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 10 client.4575.objecter successfully canceled tick 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 >> v1:172.21.15.134:6800/610837766 conn(0x7fe9b406e110 legacy=0x7fe9b40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 10 monclient: shutdown 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 >> v1:172.21.15.134:6789/0 conn(0x7fe9c81430c0 legacy=0x7fe9c81d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 shutdown_connections 2024-03-25T17:37:39.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 >> 172.21.15.134:0/1851915382 conn(0x7fe9c80b7ea0 msgr2=0x7fe9c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 shutdown_connections 2024-03-25T17:37:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 1 -- 172.21.15.134:0/1851915382 wait complete. 2024-03-25T17:37:39.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:39.461+0000 7fe9cdac9640 1 librados: shutdown 2024-03-25T17:37:39.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:46.281 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.397+0000 7f89039bc640 10 monclient: get_monmap_and_config 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.397+0000 7f89039bc640 10 monclient: build_initial_monmap 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.397+0000 7f89039bc640 10 monclient: monmap: 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:46.398295+0000 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:46.398295+0000 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:46.398 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:46.399 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:46.399 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:46.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.398+0000 7f89039bc640 1 Processor -- start 2024-03-25T17:37:46.399 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.398+0000 7f89039bc640 1 -- start start 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.398+0000 7f89039bc640 10 monclient: init 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.398+0000 7f89039bc640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.398+0000 7f89039bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.398+0000 7f89039bc640 10 monclient(hunting): picked mon.noname-c con 0x7f88fc0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.399+0000 7f89039bc640 10 monclient(hunting): picked mon.noname-b con 0x7f88fc13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.399+0000 7f89039bc640 10 monclient(hunting): picked mon.noname-a con 0x7f88fc1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.399+0000 7f89039bc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc142520 con 0x7f88fc1430c0 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.399+0000 7f89039bc640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc0b28a0 con 0x7f88fc0a98f0 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.399+0000 7f89039bc640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc0b2a30 con 0x7f88fc13f710 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.399+0000 7f89039bc640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:46.400 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.399+0000 7f89039bc640 10 monclient(hunting): authenticate will time out at 5819.661829s 2024-03-25T17:37:46.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.413+0000 7f8901758640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f88fc0a98f0 0x7f88fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34390/0 (socket says 172.21.15.134:34390) 2024-03-25T17:37:46.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.413+0000 7f8901758640 1 -- 172.21.15.134:0/1308256992 learned_addr learned my addr 172.21.15.134:0/1308256992 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.430+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202627395 0 0) 0x7f88fc0b28a0 con 0x7f88fc0a98f0 2024-03-25T17:37:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.430+0000 7f88f3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.430+0000 7f88f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.430+0000 7f88f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.430+0000 7f88f3fff640 10 monclient(hunting): my global_id is 4609 2024-03-25T17:37:46.431 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.430+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e8003520 con 0x7f88fc0a98f0 2024-03-25T17:37:46.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.432+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140659657 0 0) 0x7f88fc0b2a30 con 0x7f88fc13f710 2024-03-25T17:37:46.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.432+0000 7f88f3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:46.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.432+0000 7f88f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:46.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.432+0000 7f88f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:46.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.432+0000 7f88f3fff640 10 monclient(hunting): my global_id is 34565 2024-03-25T17:37:46.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.432+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88fc0b28a0 con 0x7f88fc13f710 2024-03-25T17:37:46.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.448+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851199276 0 0) 0x7f88fc142520 con 0x7f88fc1430c0 2024-03-25T17:37:46.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.448+0000 7f88f3fff640 10 monclient(hunting): _negotiate 2024-03-25T17:37:46.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.448+0000 7f88f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:46.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.448+0000 7f88f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:46.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.448+0000 7f88f3fff640 10 monclient(hunting): my global_id is 4578 2024-03-25T17:37:46.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.448+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88fc0b2a30 con 0x7f88fc1430c0 2024-03-25T17:37:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2854545455 0 0) 0x7f88e8003520 con 0x7f88fc0a98f0 2024-03-25T17:37:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 >> v1:172.21.15.134:6790/0 conn(0x7f88fc13f710 legacy=0x7f88fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 >> v1:172.21.15.134:6789/0 conn(0x7f88fc1430c0 legacy=0x7f88fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 10 monclient: _finish_hunting 0 2024-03-25T17:37:46.453 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 1 monclient: found mon.noname-c 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0b44c0 con 0x7f88fc0a98f0 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 10 monclient: _finish_auth 0 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 10 monclient: _check_auth_tickets 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f88f8001690 con 0x7f88fc0a98f0 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f88f3fff640 10 monclient: dump: 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.454 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f89039bc640 5 monclient: authenticate success, global_id 4609 2024-03-25T17:37:46.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.452+0000 7f89039bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:46.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88f80018b0 con 0x7f88fc0a98f0 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f88f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f88f3fff640 1 -- 172.21.15.134:0/1308256992 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f88f8001c40 con 0x7f88fc0a98f0 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f88f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f88f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f88f3fff640 10 monclient: dump: 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:46.467 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:46.468 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:46.468 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f89039bc640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:46.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.466+0000 7f89039bc640 4 set_mon_vals no callback set 2024-03-25T17:37:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.467+0000 7f89039bc640 10 monclient: shutdown 2024-03-25T17:37:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.467+0000 7f89039bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.467+0000 7f89039bc640 1 -- 172.21.15.134:0/1308256992 >> v1:172.21.15.159:6789/0 conn(0x7f88fc0a98f0 legacy=0x7f88fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.467+0000 7f89039bc640 1 -- 172.21.15.134:0/1308256992 shutdown_connections 2024-03-25T17:37:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.467+0000 7f89039bc640 1 -- 172.21.15.134:0/1308256992 >> 172.21.15.134:0/1308256992 conn(0x7f88fc0b7ea0 msgr2=0x7f88fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:46.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.467+0000 7f89039bc640 1 -- 172.21.15.134:0/1308256992 shutdown_connections 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.467+0000 7f89039bc640 1 -- 172.21.15.134:0/1308256992 wait complete. 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 10 monclient: build_initial_monmap 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 10 monclient: monmap: 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:46.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 1 librados: starting msgr at 2024-03-25T17:37:46.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 1 librados: starting objecter 2024-03-25T17:37:46.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 1 Processor -- start 2024-03-25T17:37:46.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 1 -- start start 2024-03-25T17:37:46.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 1 librados: setting wanted keys 2024-03-25T17:37:46.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 1 librados: calling monclient init 2024-03-25T17:37:46.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.468+0000 7f89039bc640 10 monclient: init 2024-03-25T17:37:46.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.469+0000 7f89039bc640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:46.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.469+0000 7f89039bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:37:46.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.469+0000 7f89039bc640 10 monclient(hunting): picked mon.noname-c con 0x7f88fc0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:46.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.469+0000 7f89039bc640 10 monclient(hunting): picked mon.noname-a con 0x7f88fc13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:46.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.469+0000 7f89039bc640 10 monclient(hunting): picked mon.noname-b con 0x7f88fc1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:46.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.470+0000 7f89039bc640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc138250 con 0x7f88fc13f710 2024-03-25T17:37:46.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.470+0000 7f89039bc640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc137d50 con 0x7f88fc1430c0 2024-03-25T17:37:46.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.470+0000 7f89039bc640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88fc137ee0 con 0x7f88fc0a98f0 2024-03-25T17:37:46.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.470+0000 7f89039bc640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:46.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.470+0000 7f89039bc640 10 monclient(hunting): authenticate will time out at 5819.732852s 2024-03-25T17:37:46.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.482+0000 7f8901f59640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f88fc1430c0 0x7f88fc1d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:34458/0 (socket says 172.21.15.134:34458) 2024-03-25T17:37:46.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.482+0000 7f8901f59640 1 -- 172.21.15.134:0/426188941 learned_addr learned my addr 172.21.15.134:0/426188941 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:46.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.483+0000 7f88f1ffb640 10 client.?.objecter ms_handle_connect 0x7f88fc1430c0 2024-03-25T17:37:46.484 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.483+0000 7f88f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:46.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.484+0000 7f88f1ffb640 10 client.?.objecter ms_handle_connect 0x7f88fc13f710 2024-03-25T17:37:46.485 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.484+0000 7f88f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:46.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.490+0000 7f88f1ffb640 10 client.?.objecter ms_handle_connect 0x7f88fc0a98f0 2024-03-25T17:37:46.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.490+0000 7f88f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:46.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.494+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007736832 0 0) 0x7f88fc138250 con 0x7f88fc13f710 2024-03-25T17:37:46.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.494+0000 7f88f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:46.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.494+0000 7f88f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:46.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.494+0000 7f88f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:46.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.494+0000 7f88f1ffb640 10 monclient(hunting): my global_id is 4581 2024-03-25T17:37:46.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.494+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e0004eb0 con 0x7f88fc13f710 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.502+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474779606 0 0) 0x7f88e0004eb0 con 0x7f88fc13f710 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.502+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88fc138250 con 0x7f88fc13f710 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.502+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f88ec001a70 con 0x7f88fc13f710 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.502+0000 7f88f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.502+0000 7f88f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.502+0000 7f88f1ffb640 10 monclient(hunting): dump: 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:46.503 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:46.504 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:46.504 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:46.504 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:46.504 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:46.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.502+0000 7f88f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:46.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397541496 0 0) 0x7f88fc137ee0 con 0x7f88fc0a98f0 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): my global_id is 34568 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e0004eb0 con 0x7f88fc0a98f0 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055179900 0 0) 0x7f88fc137d50 con 0x7f88fc1430c0 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 10 monclient(hunting): my global_id is 4612 2024-03-25T17:37:46.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.503+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88fc137ee0 con 0x7f88fc1430c0 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.511+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3820269871 0 0) 0x7f88fc138250 con 0x7f88fc13f710 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.511+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 >> v1:172.21.15.134:6790/0 conn(0x7f88fc0a98f0 legacy=0x7f88fc137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.511+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 >> v1:172.21.15.159:6789/0 conn(0x7f88fc1430c0 legacy=0x7f88fc1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.511+0000 7f88f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.511+0000 7f88f1ffb640 1 monclient: found mon.a 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.511+0000 7f88f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f88f1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0b7730 con 0x7f88fc13f710 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f88f1ffb640 10 monclient: _finish_auth 0 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f88f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f88f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:46.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 5 monclient: authenticate success, global_id 4581 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 10 monclient: _renew_subs 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc0b7920 con 0x7f88fc13f710 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 10 monclient: _renew_subs 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc1e33d0 con 0x7f88fc13f710 2024-03-25T17:37:46.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.512+0000 7f89039bc640 1 librados: init done 2024-03-25T17:37:46.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.513+0000 7f89039bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:46.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.513+0000 7f89039bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:46.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.513+0000 7f89039bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:46.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.513+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc0627b0 con 0x7f88fc13f710 2024-03-25T17:37:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.798+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88ec001bd0 con 0x7f88fc13f710 2024-03-25T17:37:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.798+0000 7f88f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.798+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f88ec003a60 con 0x7f88fc13f710 2024-03-25T17:37:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.798+0000 7f88f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:46.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.798+0000 7f88f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.798+0000 7f88f1ffb640 10 monclient: dump: 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.798+0000 7f88f37fe640 4 set_mon_vals no callback set 2024-03-25T17:37:46.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.799+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f88ec019a30 con 0x7f88fc13f710 2024-03-25T17:37:46.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.800+0000 7f8901758640 10 monclient: get_auth_request con 0x7f88e006ddb0 auth_method 0 2024-03-25T17:37:46.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.800+0000 7f88f1ffb640 10 client.4581.objecter ms_handle_connect 0x7f88e006ddb0 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f88ec053670 con 0x7f88fc13f710 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 10 client.4581.objecter ms_dispatch 0x7f88fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88ec0b08b0 con 0x7f88fc13f710 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:46.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:46.805+0000 7f88f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:47.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.028+0000 7f89039bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:47.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.028+0000 7f89039bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:47.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.028+0000 7f89039bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:37:47.029 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.028+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88fc138400 con 0x7f88fc13f710 2024-03-25T17:37:47.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.048+0000 7f88f1ffb640 1 -- 172.21.15.134:0/426188941 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f88fc138400 con 0x7f88fc13f710 2024-03-25T17:37:47.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.048+0000 7f88f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:47.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.048+0000 7f88f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:47.049 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:47.049 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 10 librados: watch_flush enter 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 10 librados: watch_flush exit 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 10 client.4581.objecter successfully canceled tick 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 >> v1:172.21.15.134:6800/610837766 conn(0x7f88e006ddb0 legacy=0x7f88e0070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 10 monclient: shutdown 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:47.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 >> v1:172.21.15.134:6789/0 conn(0x7f88fc13f710 legacy=0x7f88fc1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 shutdown_connections 2024-03-25T17:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 >> 172.21.15.134:0/426188941 conn(0x7f88fc0b7ea0 msgr2=0x7f88fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 shutdown_connections 2024-03-25T17:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 1 -- 172.21.15.134:0/426188941 wait complete. 2024-03-25T17:37:47.053 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:47.051+0000 7f89039bc640 1 librados: shutdown 2024-03-25T17:37:47.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:37:50.549 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:55.565 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.550 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:55.579 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.560 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:55.576 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.561 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:55.589 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.588 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:08:55.604 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.595 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:08:55.624 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.611 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:55.627 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.627 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:55.644 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:50.663 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:08:55.679 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:37:53.871 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:37:53.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.986+0000 7f7622aa5640 10 monclient: get_monmap_and_config 2024-03-25T17:37:53.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.986+0000 7f7622aa5640 10 monclient: build_initial_monmap 2024-03-25T17:37:53.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.986+0000 7f7622aa5640 10 monclient: monmap: 2024-03-25T17:37:53.987 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:53.987 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:37:53.987831+0000 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:37:53.987831+0000 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:37:53.988 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:53.989 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.987+0000 7f7622aa5640 1 Processor -- start 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.987+0000 7f7622aa5640 1 -- start start 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.987+0000 7f7622aa5640 10 monclient: init 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f761c06adf0 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f761c0aee10 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f761c140860 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0af690 con 0x7f761c0aee10 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0b0c70 con 0x7f761c06adf0 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c0b0f50 con 0x7f761c140860 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:53.990 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.988+0000 7f7622aa5640 10 monclient(hunting): authenticate will time out at 5827.251744s 2024-03-25T17:37:53.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.997+0000 7f7621042640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f761c140860 0x7f761c14b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50556/0 (socket says 172.21.15.134:50556) 2024-03-25T17:37:53.997 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:53.997+0000 7f7621042640 1 -- 172.21.15.134:0/1316004188 learned_addr learned my addr 172.21.15.134:0/1316004188 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:54.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.009+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234396448 0 0) 0x7f761c0b0f50 con 0x7f761c140860 2024-03-25T17:37:54.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.009+0000 7f761b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:54.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.009+0000 7f761b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:54.010 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.009+0000 7f761b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:54.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.009+0000 7f761b7fe640 10 monclient(hunting): my global_id is 34571 2024-03-25T17:37:54.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.010+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f760c003520 con 0x7f761c140860 2024-03-25T17:37:54.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.011+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41373271 0 0) 0x7f761c0af690 con 0x7f761c0aee10 2024-03-25T17:37:54.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.011+0000 7f761b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:54.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.011+0000 7f761b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:54.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.011+0000 7f761b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:54.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.011+0000 7f761b7fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T17:37:54.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.011+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c0b0f50 con 0x7f761c0aee10 2024-03-25T17:37:54.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.014+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212438498 0 0) 0x7f761c0b0c70 con 0x7f761c06adf0 2024-03-25T17:37:54.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.014+0000 7f761b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:37:54.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.014+0000 7f761b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:54.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.014+0000 7f761b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:54.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.014+0000 7f761b7fe640 10 monclient(hunting): my global_id is 4615 2024-03-25T17:37:54.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.014+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c0af690 con 0x7f761c06adf0 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476241826 0 0) 0x7f761c0b0f50 con 0x7f761c0aee10 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 >> v1:172.21.15.134:6790/0 conn(0x7f761c140860 legacy=0x7f761c14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 >> v1:172.21.15.159:6789/0 conn(0x7f761c06adf0 legacy=0x7f761c13d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 1 monclient: found mon.noname-a 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c13dc40 con 0x7f761c0aee10 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:37:54.021 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7608001670 con 0x7f761c0aee10 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f761b7fe640 10 monclient: dump: 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:54.022 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:54.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f7622aa5640 5 monclient: authenticate success, global_id 4584 2024-03-25T17:37:54.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.020+0000 7f7622aa5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:37:54.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76080017d0 con 0x7f761c0aee10 2024-03-25T17:37:54.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f761b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:54.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f761b7fe640 1 -- 172.21.15.134:0/1316004188 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7608001be0 con 0x7f761c0aee10 2024-03-25T17:37:54.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f761b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f761b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f761b7fe640 10 monclient: dump: 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:54.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f7622aa5640 10 monclient: get_monmap_and_config success 2024-03-25T17:37:54.038 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.035+0000 7f7622aa5640 4 set_mon_vals no callback set 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.036+0000 7f7622aa5640 10 monclient: shutdown 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.036+0000 7f7622aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.036+0000 7f7622aa5640 1 -- 172.21.15.134:0/1316004188 >> v1:172.21.15.134:6789/0 conn(0x7f761c0aee10 legacy=0x7f761c140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.036+0000 7f7622aa5640 1 -- 172.21.15.134:0/1316004188 shutdown_connections 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.036+0000 7f7622aa5640 1 -- 172.21.15.134:0/1316004188 >> 172.21.15.134:0/1316004188 conn(0x7f761c0b4a80 msgr2=0x7f761c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.036+0000 7f7622aa5640 1 -- 172.21.15.134:0/1316004188 shutdown_connections 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.036+0000 7f7622aa5640 1 -- 172.21.15.134:0/1316004188 wait complete. 2024-03-25T17:37:54.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.037+0000 7f7622aa5640 10 monclient: build_initial_monmap 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.037+0000 7f7622aa5640 10 monclient: monmap: 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.038+0000 7f7622aa5640 1 librados: starting msgr at 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.038+0000 7f7622aa5640 1 librados: starting objecter 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.038+0000 7f7622aa5640 1 Processor -- start 2024-03-25T17:37:54.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.038+0000 7f7622aa5640 1 -- start start 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.038+0000 7f7622aa5640 1 librados: setting wanted keys 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.038+0000 7f7622aa5640 1 librados: calling monclient init 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.038+0000 7f7622aa5640 10 monclient: init 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 10 monclient: _reopen_session rank -1 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 10 monclient(hunting): picked mon.noname-a con 0x7f761c06adf0 addr v1:172.21.15.134:6789/0 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 10 monclient(hunting): picked mon.noname-c con 0x7f761c0aee10 addr v1:172.21.15.134:6790/0 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 10 monclient(hunting): picked mon.noname-b con 0x7f761c141430 addr v1:172.21.15.159:6789/0 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c1376f0 con 0x7f761c06adf0 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c1424b0 con 0x7f761c141430 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761c131430 con 0x7f761c0aee10 2024-03-25T17:37:54.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 10 monclient(hunting): _renew_subs 2024-03-25T17:37:54.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.039+0000 7f7622aa5640 10 monclient(hunting): authenticate will time out at 5827.302448s 2024-03-25T17:37:54.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.045+0000 7f761bfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f761c0aee10 0x7f761c1da9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:50592/0 (socket says 172.21.15.134:50592) 2024-03-25T17:37:54.046 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.046+0000 7f761bfff640 1 -- 172.21.15.134:0/2565203201 learned_addr learned my addr 172.21.15.134:0/2565203201 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:37:54.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.050+0000 7f76197fa640 10 client.?.objecter ms_handle_connect 0x7f761c0aee10 2024-03-25T17:37:54.050 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.050+0000 7f76197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:54.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.051+0000 7f76197fa640 10 client.?.objecter ms_handle_connect 0x7f761c141430 2024-03-25T17:37:54.051 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.051+0000 7f76197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:54.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.060+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201379794 0 0) 0x7f761c131430 con 0x7f761c0aee10 2024-03-25T17:37:54.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.060+0000 7f76197fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:54.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.060+0000 7f76197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:54.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.060+0000 7f76197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:54.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.060+0000 7f76197fa640 10 monclient(hunting): my global_id is 34574 2024-03-25T17:37:54.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.060+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7600004eb0 con 0x7f761c0aee10 2024-03-25T17:37:54.067 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.067+0000 7f76197fa640 10 client.?.objecter ms_handle_connect 0x7f761c06adf0 2024-03-25T17:37:54.068 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.067+0000 7f76197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:37:54.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.069+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949397562 0 0) 0x7f761c1424b0 con 0x7f761c141430 2024-03-25T17:37:54.069 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.069+0000 7f76197fa640 10 monclient(hunting): _negotiate 2024-03-25T17:37:54.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.069+0000 7f76197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:37:54.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.069+0000 7f76197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:37:54.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.069+0000 7f76197fa640 10 monclient(hunting): my global_id is 4618 2024-03-25T17:37:54.070 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.069+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f761c131430 con 0x7f761c141430 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.073+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3527178794 0 0) 0x7f7600004eb0 con 0x7f761c0aee10 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.073+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f761c1424b0 con 0x7f761c0aee10 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.073+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7608001670 con 0x7f761c0aee10 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.073+0000 7f76197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.073+0000 7f76197fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.073+0000 7f76197fa640 10 monclient(hunting): dump: 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.074 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.075 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:54.075 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:54.075 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:54.075 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:54.075 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:54.075 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:54.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.073+0000 7f76197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4005748173 0 0) 0x7f761c1424b0 con 0x7f761c0aee10 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 >> v1:172.21.15.159:6789/0 conn(0x7f761c141430 legacy=0x7f761c130d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 >> v1:172.21.15.134:6789/0 conn(0x7f761c06adf0 legacy=0x7f761c06ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 10 monclient: _finish_hunting 0 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 1 monclient: found mon.c 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761c1315c0 con 0x7f761c0aee10 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 10 monclient: _finish_auth 0 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 10 monclient: _check_auth_tickets 2024-03-25T17:37:54.090 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f76197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 5 monclient: authenticate success, global_id 34574 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 10 monclient: _renew_subs 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c070930 con 0x7f761c0aee10 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 10 client.34574.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 10 monclient: _renew_subs 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c136f90 con 0x7f761c0aee10 2024-03-25T17:37:54.091 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.089+0000 7f7622aa5640 1 librados: init done 2024-03-25T17:37:54.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.090+0000 7f7622aa5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:54.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.090+0000 7f7622aa5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:54.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.090+0000 7f7622aa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:54.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.090+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c0627b0 con 0x7f761c0aee10 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.404+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76080039c0 con 0x7f761c0aee10 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.404+0000 7f76197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.404+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f7608003d90 con 0x7f761c0aee10 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.405+0000 7f76197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.405+0000 7f76197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.405+0000 7f76197fa640 10 monclient: dump: 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:37:54.406 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.405+0000 7f761affd640 4 set_mon_vals no callback set 2024-03-25T17:37:54.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.405+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f7608003fb0 con 0x7f761c0aee10 2024-03-25T17:37:54.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.406+0000 7f7620841640 10 monclient: get_auth_request con 0x7f760006db60 auth_method 0 2024-03-25T17:37:54.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.406+0000 7f76197fa640 10 client.34574.objecter ms_handle_connect 0x7f760006db60 2024-03-25T17:37:54.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f7608054a00 con 0x7f761c0aee10 2024-03-25T17:37:54.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 10 client.34574.objecter ms_dispatch 0x7f761c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:37:54.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 3 client.34574.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:37:54.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 3 client.34574.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:37:54.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 20 client.34574.objecter dump_active .. 0 homeless 2024-03-25T17:37:54.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7608050510 con 0x7f761c0aee10 2024-03-25T17:37:54.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:37:54.413 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.411+0000 7f76197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:37:54.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.661+0000 7f7622aa5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:54.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.661+0000 7f7622aa5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:54.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.661+0000 7f7622aa5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:37:54.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.661+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f761c0aff00 con 0x7f761c0aee10 2024-03-25T17:37:54.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.698+0000 7f76197fa640 1 -- 172.21.15.134:0/2565203201 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f7608052440 con 0x7f761c0aee10 2024-03-25T17:37:54.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.698+0000 7f76197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:37:54.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.698+0000 7f76197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:37:54.700 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:37:54.700 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:37:54.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 10 librados: watch_flush enter 2024-03-25T17:37:54.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 10 librados: watch_flush exit 2024-03-25T17:37:54.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 20 client.34574.objecter shutdown clearing up homeless session... 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 10 client.34574.objecter successfully canceled tick 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 >> v1:172.21.15.134:6800/610837766 conn(0x7f760006db60 legacy=0x7f7600070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 10 monclient: shutdown 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 >> v1:172.21.15.134:6790/0 conn(0x7f761c0aee10 legacy=0x7f761c1da9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 shutdown_connections 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 >> 172.21.15.134:0/2565203201 conn(0x7f761c0b4a80 msgr2=0x7f761c13a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 shutdown_connections 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 1 -- 172.21.15.134:0/2565203201 wait complete. 2024-03-25T17:37:54.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:37:54.701+0000 7f7622aa5640 1 librados: shutdown 2024-03-25T17:37:54.714 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:01.521 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:01.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.635+0000 7fa8b777b640 10 monclient: get_monmap_and_config 2024-03-25T17:38:01.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.635+0000 7fa8b777b640 10 monclient: build_initial_monmap 2024-03-25T17:38:01.636 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.635+0000 7fa8b777b640 10 monclient: monmap: 2024-03-25T17:38:01.636 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:01.636 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:01.636 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:01.636552+0000 2024-03-25T17:38:01.636 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:01.636552+0000 2024-03-25T17:38:01.637 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:01.637 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:01.637 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:01.637 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:01.637 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:01.637 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.636+0000 7fa8b777b640 1 Processor -- start 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.636+0000 7fa8b777b640 1 -- start start 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.636+0000 7fa8b777b640 10 monclient: init 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b00b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b00b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:01.639 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b014be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:01.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00af940 con 0x7fa8b00b4d60 2024-03-25T17:38:01.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00b0a70 con 0x7fa8b00b1730 2024-03-25T17:38:01.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00b0d50 con 0x7fa8b014be90 2024-03-25T17:38:01.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:01.640 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.637+0000 7fa8b777b640 10 monclient(hunting): authenticate will time out at 5834.899937s 2024-03-25T17:38:01.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.644+0000 7fa8b5d18640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fa8b014be90 0x7fa8b014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:51198/0 (socket says 172.21.15.134:51198) 2024-03-25T17:38:01.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.644+0000 7fa8b5d18640 1 -- 172.21.15.134:0/3964876327 learned_addr learned my addr 172.21.15.134:0/3964876327 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:01.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.660+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594954848 0 0) 0x7fa8b00b0d50 con 0x7fa8b014be90 2024-03-25T17:38:01.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.660+0000 7fa8a7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:01.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.660+0000 7fa8a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:01.661 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.660+0000 7fa8a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:01.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.660+0000 7fa8a7fff640 10 monclient(hunting): my global_id is 34577 2024-03-25T17:38:01.662 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.660+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89c003520 con 0x7fa8b014be90 2024-03-25T17:38:01.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.667+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253590078 0 0) 0x7fa8b00b0a70 con 0x7fa8b00b1730 2024-03-25T17:38:01.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.667+0000 7fa8a7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:01.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.667+0000 7fa8a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:01.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.667+0000 7fa8a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:01.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.667+0000 7fa8a7fff640 10 monclient(hunting): my global_id is 4621 2024-03-25T17:38:01.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.667+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b00b0d50 con 0x7fa8b00b1730 2024-03-25T17:38:01.670 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1271526187 0 0) 0x7fa89c003520 con 0x7fa8b014be90 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 >> v1:172.21.15.159:6789/0 conn(0x7fa8b00b1730 legacy=0x7fa8b00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 >> v1:172.21.15.134:6789/0 conn(0x7fa8b00b4d60 legacy=0x7fa8b013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 1 monclient: found mon.noname-b 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b00b1030 con 0x7fa8b014be90 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 10 monclient: _finish_auth 0 2024-03-25T17:38:01.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa8a8001a10 con 0x7fa8b014be90 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8a7fff640 10 monclient: dump: 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:01.672 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:01.673 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:01.673 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:01.673 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:01.673 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:01.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8b777b640 5 monclient: authenticate success, global_id 34577 2024-03-25T17:38:01.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.669+0000 7fa8b777b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8a8001b70 con 0x7fa8b014be90 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8a7fff640 1 -- 172.21.15.134:0/3964876327 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa8a8001f80 con 0x7fa8b014be90 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8a7fff640 10 monclient: dump: 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:01.694 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8b777b640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:01.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.693+0000 7fa8b777b640 4 set_mon_vals no callback set 2024-03-25T17:38:01.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.694+0000 7fa8b777b640 10 monclient: shutdown 2024-03-25T17:38:01.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.694+0000 7fa8b777b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:01.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.694+0000 7fa8b777b640 1 -- 172.21.15.134:0/3964876327 >> v1:172.21.15.134:6790/0 conn(0x7fa8b014be90 legacy=0x7fa8b014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:01.698 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.694+0000 7fa8b777b640 1 -- 172.21.15.134:0/3964876327 shutdown_connections 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.694+0000 7fa8b777b640 1 -- 172.21.15.134:0/3964876327 >> 172.21.15.134:0/3964876327 conn(0x7fa8b00617a0 msgr2=0x7fa8b0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.694+0000 7fa8b777b640 1 -- 172.21.15.134:0/3964876327 shutdown_connections 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.694+0000 7fa8b777b640 1 -- 172.21.15.134:0/3964876327 wait complete. 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 10 monclient: build_initial_monmap 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 10 monclient: monmap: 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:01.699 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 1 librados: starting msgr at 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 1 librados: starting objecter 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 1 Processor -- start 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 1 -- start start 2024-03-25T17:38:01.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 1 librados: setting wanted keys 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 1 librados: calling monclient init 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.695+0000 7fa8b777b640 10 monclient: init 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b00b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b00b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b014be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b01d8600 con 0x7fa8b00b4d60 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b01d79d0 con 0x7fa8b00b1730 2024-03-25T17:38:01.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b01434e0 con 0x7fa8b014be90 2024-03-25T17:38:01.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:01.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.696+0000 7fa8b777b640 10 monclient(hunting): authenticate will time out at 5834.959288s 2024-03-25T17:38:01.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.702+0000 7fa8b5d18640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fa8b014be90 0x7fa8b013b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:51260/0 (socket says 172.21.15.134:51260) 2024-03-25T17:38:01.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.702+0000 7fa8b5d18640 1 -- 172.21.15.134:0/3229544033 learned_addr learned my addr 172.21.15.134:0/3229544033 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:01.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.709+0000 7fa8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa8b014be90 2024-03-25T17:38:01.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.709+0000 7fa8a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:01.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.710+0000 7fa8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa8b00b4d60 2024-03-25T17:38:01.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.710+0000 7fa8a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:01.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.715+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073047040 0 0) 0x7fa8b01d8600 con 0x7fa8b00b4d60 2024-03-25T17:38:01.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.715+0000 7fa8a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:01.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.715+0000 7fa8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:01.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.715+0000 7fa8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:01.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.715+0000 7fa8a5ffb640 10 monclient(hunting): my global_id is 4593 2024-03-25T17:38:01.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.715+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa894004eb0 con 0x7fa8b00b4d60 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2546042334 0 0) 0x7fa8b01434e0 con 0x7fa8b014be90 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 10 monclient(hunting): my global_id is 34580 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b01d8600 con 0x7fa8b014be90 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 10 client.?.objecter ms_handle_connect 0x7fa8b00b1730 2024-03-25T17:38:01.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.722+0000 7fa8a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:01.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.729+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1251816148 0 0) 0x7fa894004eb0 con 0x7fa8b00b4d60 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.729+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8b01434e0 con 0x7fa8b00b4d60 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.729+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa8a0001930 con 0x7fa8b00b4d60 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.729+0000 7fa8a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.729+0000 7fa8a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.729+0000 7fa8a5ffb640 10 monclient(hunting): dump: 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:01.731 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:01.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.729+0000 7fa8a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:01.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.743+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1714820048 0 0) 0x7fa8b01d8600 con 0x7fa8b014be90 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.743+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa894004eb0 con 0x7fa8b014be90 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.743+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa8a8001f40 con 0x7fa8b014be90 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.743+0000 7fa8a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.743+0000 7fa8a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.743+0000 7fa8a5ffb640 10 monclient(hunting): dump: 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:01.745 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:01.746 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:01.746 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:01.746 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:01.746 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:01.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.747+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2160192042 0 0) 0x7fa8b01d79d0 con 0x7fa8b00b1730 2024-03-25T17:38:01.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.747+0000 7fa8a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:01.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.747+0000 7fa8a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:01.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.747+0000 7fa8a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:01.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.747+0000 7fa8a5ffb640 10 monclient(hunting): my global_id is 4624 2024-03-25T17:38:01.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.747+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b01d8600 con 0x7fa8b00b1730 2024-03-25T17:38:01.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1262150499 0 0) 0x7fa8b01434e0 con 0x7fa8b00b4d60 2024-03-25T17:38:01.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 >> v1:172.21.15.134:6790/0 conn(0x7fa8b014be90 legacy=0x7fa8b013b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:01.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 >> v1:172.21.15.159:6789/0 conn(0x7fa8b00b1730 legacy=0x7fa8b0139290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:01.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:38:01.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 1 monclient: found mon.a 2024-03-25T17:38:01.753 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b0143670 con 0x7fa8b00b4d60 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8b777b640 5 monclient: authenticate success, global_id 4593 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8b777b640 10 monclient: _renew_subs 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8b777b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.751+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b0142b60 con 0x7fa8b00b4d60 2024-03-25T17:38:01.754 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.752+0000 7fa8b777b640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:01.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.752+0000 7fa8b777b640 10 monclient: _renew_subs 2024-03-25T17:38:01.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.752+0000 7fa8b777b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:01.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.752+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b013bfe0 con 0x7fa8b00b4d60 2024-03-25T17:38:01.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.752+0000 7fa8b777b640 1 librados: init done 2024-03-25T17:38:01.755 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.753+0000 7fa8b777b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:01.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.753+0000 7fa8b777b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:01.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.753+0000 7fa8b777b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:01.756 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:01.753+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b01d77c0 con 0x7fa8b00b4d60 2024-03-25T17:38:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.062+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8a0003610 con 0x7fa8b00b4d60 2024-03-25T17:38:02.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.062+0000 7fa8a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.062+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fa8a0003920 con 0x7fa8b00b4d60 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.062+0000 7fa8a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.062+0000 7fa8a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.062+0000 7fa8a5ffb640 10 monclient: dump: 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:02.064 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:02.065 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:02.065 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:02.065 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:02.065 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:02.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.062+0000 7fa8a77fe640 4 set_mon_vals no callback set 2024-03-25T17:38:02.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.063+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fa8a00198f0 con 0x7fa8b00b4d60 2024-03-25T17:38:02.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.064+0000 7fa8b5517640 10 monclient: get_auth_request con 0x7fa89406df50 auth_method 0 2024-03-25T17:38:02.066 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.064+0000 7fa8a5ffb640 10 client.4593.objecter ms_handle_connect 0x7fa89406df50 2024-03-25T17:38:02.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fa8a0053590 con 0x7fa8b00b4d60 2024-03-25T17:38:02.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 10 client.4593.objecter ms_dispatch 0x7fa8b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:02.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 3 client.4593.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:02.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 3 client.4593.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:02.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-25T17:38:02.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8a004f0a0 con 0x7fa8b00b4d60 2024-03-25T17:38:02.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:02.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.070+0000 7fa8a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:02.296 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.295+0000 7fa8b777b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:02.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.295+0000 7fa8b777b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:02.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.295+0000 7fa8b777b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:02.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.296+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8b01d7b60 con 0x7fa8b00b4d60 2024-03-25T17:38:02.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.320+0000 7fa8a5ffb640 1 -- 172.21.15.134:0/3229544033 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fa8a0050fd0 con 0x7fa8b00b4d60 2024-03-25T17:38:02.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.320+0000 7fa8a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:02.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.320+0000 7fa8a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:02.322 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:02.322 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 10 librados: watch_flush enter 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 10 librados: watch_flush exit 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 10 client.4593.objecter successfully canceled tick 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 >> v1:172.21.15.134:6800/610837766 conn(0x7fa89406df50 legacy=0x7fa894070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 10 monclient: shutdown 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:02.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.323+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 >> v1:172.21.15.134:6789/0 conn(0x7fa8b00b4d60 legacy=0x7fa8b01d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:02.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.324+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 shutdown_connections 2024-03-25T17:38:02.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.324+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 >> 172.21.15.134:0/3229544033 conn(0x7fa8b00617a0 msgr2=0x7fa8b00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:02.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.324+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 shutdown_connections 2024-03-25T17:38:02.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.324+0000 7fa8b777b640 1 -- 172.21.15.134:0/3229544033 wait complete. 2024-03-25T17:38:02.326 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:02.324+0000 7fa8b777b640 1 librados: shutdown 2024-03-25T17:38:02.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:09.144 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:09.257 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.255+0000 7f83a015d640 10 monclient: get_monmap_and_config 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.255+0000 7f83a015d640 10 monclient: build_initial_monmap 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.255+0000 7f83a015d640 10 monclient: monmap: 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:09.257673+0000 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:09.257673+0000 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:09.258 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:09.259 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:09.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.256+0000 7f83a015d640 1 Processor -- start 2024-03-25T17:38:09.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.256+0000 7f83a015d640 1 -- start start 2024-03-25T17:38:09.259 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.256+0000 7f83a015d640 10 monclient: init 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 10 monclient(hunting): picked mon.noname-a con 0x7f83980a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 10 monclient(hunting): picked mon.noname-b con 0x7f839813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 10 monclient(hunting): picked mon.noname-c con 0x7f83981430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8398142520 con 0x7f83980a98f0 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83980b28a0 con 0x7f83981430c0 2024-03-25T17:38:09.260 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83980b2a30 con 0x7f839813f710 2024-03-25T17:38:09.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:09.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.257+0000 7f83a015d640 10 monclient(hunting): authenticate will time out at 5842.521102s 2024-03-25T17:38:09.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.260+0000 7f839def9640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f83980a98f0 0x7f83980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:52728/0 (socket says 172.21.15.134:52728) 2024-03-25T17:38:09.261 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.260+0000 7f839def9640 1 -- 172.21.15.134:0/2823941973 learned_addr learned my addr 172.21.15.134:0/2823941973 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:09.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.269+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 483504544 0 0) 0x7f8398142520 con 0x7f83980a98f0 2024-03-25T17:38:09.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.269+0000 7f839cef7640 10 monclient(hunting): _negotiate 2024-03-25T17:38:09.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.269+0000 7f839cef7640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:09.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.269+0000 7f839cef7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:09.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.269+0000 7f839cef7640 10 monclient(hunting): my global_id is 4596 2024-03-25T17:38:09.271 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.269+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8384003520 con 0x7f83980a98f0 2024-03-25T17:38:09.278 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450415515 0 0) 0x7f8384003520 con 0x7f83980a98f0 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 >> v1:172.21.15.134:6790/0 conn(0x7f839813f710 legacy=0x7f83981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 >> v1:172.21.15.159:6789/0 conn(0x7f83981430c0 legacy=0x7f839814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 10 monclient: _finish_hunting 0 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 1 monclient: found mon.noname-a 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980b44c0 con 0x7f83980a98f0 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 10 monclient: _finish_auth 0 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 10 monclient: _check_auth_tickets 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8394001690 con 0x7f83980a98f0 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:09.279 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f839cef7640 10 monclient: dump: 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f83a015d640 5 monclient: authenticate success, global_id 4596 2024-03-25T17:38:09.280 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.277+0000 7f83a015d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:09.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8394001860 con 0x7f83980a98f0 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f839cef7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f839cef7640 1 -- 172.21.15.134:0/2823941973 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8394001cb0 con 0x7f83980a98f0 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f83a015d640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f83a015d640 4 set_mon_vals no callback set 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f839cef7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f839cef7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f839cef7640 10 monclient: dump: 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:09.285 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:09.286 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:09.286 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:09.286 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:09.286 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f83a015d640 10 monclient: shutdown 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f83a015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f83a015d640 1 -- 172.21.15.134:0/2823941973 >> v1:172.21.15.134:6789/0 conn(0x7f83980a98f0 legacy=0x7f83980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.283+0000 7f83a015d640 1 -- 172.21.15.134:0/2823941973 shutdown_connections 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.284+0000 7f83a015d640 1 -- 172.21.15.134:0/2823941973 >> 172.21.15.134:0/2823941973 conn(0x7f83980b7ea0 msgr2=0x7f83980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.284+0000 7f83a015d640 1 -- 172.21.15.134:0/2823941973 shutdown_connections 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.284+0000 7f83a015d640 1 -- 172.21.15.134:0/2823941973 wait complete. 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.284+0000 7f83a015d640 10 monclient: build_initial_monmap 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.284+0000 7f83a015d640 10 monclient: monmap: 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:09.288 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.284+0000 7f83a015d640 1 librados: starting msgr at 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 librados: starting objecter 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 Processor -- start 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 -- start start 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 librados: setting wanted keys 2024-03-25T17:38:09.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 librados: calling monclient init 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient: init 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient(hunting): picked mon.noname-c con 0x7f83980a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient(hunting): picked mon.noname-b con 0x7f839813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient(hunting): picked mon.noname-a con 0x7f83981430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8398136f30 con 0x7f83981430c0 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8398135570 con 0x7f839813f710 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8398135070 con 0x7f83980a98f0 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:09.290 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.285+0000 7f83a015d640 10 monclient(hunting): authenticate will time out at 5842.549466s 2024-03-25T17:38:09.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.296+0000 7f839def9640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f83980a98f0 0x7f83981d4790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:56022/0 (socket says 172.21.15.134:56022) 2024-03-25T17:38:09.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.296+0000 7f839def9640 1 -- 172.21.15.134:0/2794904783 learned_addr learned my addr 172.21.15.134:0/2794904783 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:09.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.300+0000 7f838e7fc640 10 client.?.objecter ms_handle_connect 0x7f839813f710 2024-03-25T17:38:09.301 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.300+0000 7f838e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:09.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.307+0000 7f838e7fc640 10 client.?.objecter ms_handle_connect 0x7f83980a98f0 2024-03-25T17:38:09.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.307+0000 7f838e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:09.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.313+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864782639 0 0) 0x7f8398135570 con 0x7f839813f710 2024-03-25T17:38:09.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.313+0000 7f838e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:09.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.313+0000 7f838e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:09.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.313+0000 7f838e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:09.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.313+0000 7f838e7fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T17:38:09.315 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.313+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f837c004eb0 con 0x7f839813f710 2024-03-25T17:38:09.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.320+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840197700 0 0) 0x7f8398135070 con 0x7f83980a98f0 2024-03-25T17:38:09.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.320+0000 7f838e7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:09.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.320+0000 7f838e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:09.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.320+0000 7f838e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:09.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.320+0000 7f838e7fc640 10 monclient(hunting): my global_id is 34583 2024-03-25T17:38:09.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.320+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8398135570 con 0x7f83980a98f0 2024-03-25T17:38:09.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.321+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4113622333 0 0) 0x7f837c004eb0 con 0x7f839813f710 2024-03-25T17:38:09.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.321+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8398135070 con 0x7f839813f710 2024-03-25T17:38:09.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.321+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8388001870 con 0x7f839813f710 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.321+0000 7f838e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.321+0000 7f838e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.321+0000 7f838e7fc640 10 monclient(hunting): dump: 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:09.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.321+0000 7f838e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:09.327 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.326+0000 7f838e7fc640 10 client.?.objecter ms_handle_connect 0x7f83981430c0 2024-03-25T17:38:09.328 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.326+0000 7f838e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:09.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 501494870 0 0) 0x7f8398135070 con 0x7f839813f710 2024-03-25T17:38:09.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 >> v1:172.21.15.134:6790/0 conn(0x7f83980a98f0 legacy=0x7f83981d4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:09.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 >> v1:172.21.15.134:6789/0 conn(0x7f83981430c0 legacy=0x7f83981db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:09.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:38:09.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 1 monclient: found mon.b 2024-03-25T17:38:09.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83981d41e0 con 0x7f839813f710 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 10 monclient: _finish_auth 0 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f838e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 5 monclient: authenticate success, global_id 4630 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 10 monclient: _renew_subs 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8398135cf0 con 0x7f839813f710 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 10 monclient: _renew_subs 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:09.337 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f839813ca50 con 0x7f839813f710 2024-03-25T17:38:09.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.334+0000 7f83a015d640 1 librados: init done 2024-03-25T17:38:09.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.335+0000 7f83a015d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:09.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.335+0000 7f83a015d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:09.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.335+0000 7f83a015d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:09.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.335+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83981d43d0 con 0x7f839813f710 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.607+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83880035f0 con 0x7f839813f710 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.607+0000 7f838e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.607+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8388001d80 con 0x7f839813f710 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.607+0000 7f838e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.607+0000 7f838e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.607+0000 7f838e7fc640 10 monclient: dump: 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:09.609 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:09.610 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.607+0000 7f838ffff640 4 set_mon_vals no callback set 2024-03-25T17:38:09.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.608+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f8388019830 con 0x7f839813f710 2024-03-25T17:38:09.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.608+0000 7f839def9640 10 monclient: get_auth_request con 0x7f837c06dd10 auth_method 0 2024-03-25T17:38:09.611 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.609+0000 7f838e7fc640 10 client.4630.objecter ms_handle_connect 0x7f837c06dd10 2024-03-25T17:38:09.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.612+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f83880535b0 con 0x7f839813f710 2024-03-25T17:38:09.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.612+0000 7f838e7fc640 10 client.4630.objecter ms_dispatch 0x7f8398000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:09.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.612+0000 7f838e7fc640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:09.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.612+0000 7f838e7fc640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:09.614 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.612+0000 7f838e7fc640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T17:38:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.614+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f838804eef0 con 0x7f839813f710 2024-03-25T17:38:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.614+0000 7f838e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:09.616 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.614+0000 7f838e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:09.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.834+0000 7f83a015d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:09.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.834+0000 7f83a015d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:09.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.834+0000 7f83a015d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:09.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.834+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83980b3ef0 con 0x7f839813f710 2024-03-25T17:38:09.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.860+0000 7f838e7fc640 1 -- 172.21.15.134:0/2794904783 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f8388050ff0 con 0x7f839813f710 2024-03-25T17:38:09.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.860+0000 7f838e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:09.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.860+0000 7f838e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:09.862 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:09.862 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 10 librados: watch_flush enter 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 10 librados: watch_flush exit 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 10 client.4630.objecter successfully canceled tick 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 >> v1:172.21.15.134:6800/610837766 conn(0x7f837c06dd10 legacy=0x7f837c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 10 monclient: shutdown 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.863+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 >> v1:172.21.15.159:6789/0 conn(0x7f839813f710 legacy=0x7f83981dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.864+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 shutdown_connections 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.864+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 >> 172.21.15.134:0/2794904783 conn(0x7f83980b7ea0 msgr2=0x7f8398074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:09.866 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.864+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 shutdown_connections 2024-03-25T17:38:09.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.864+0000 7f83a015d640 1 -- 172.21.15.134:0/2794904783 wait complete. 2024-03-25T17:38:09.867 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:09.864+0000 7f83a015d640 1 librados: shutdown 2024-03-25T17:38:09.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:10.586 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:15.602 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.637 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:15.653 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.677 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:15.694 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.679 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:15.707 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.688 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:15.704 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.695 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:15.724 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.704 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:15.721 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.735 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:15.764 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:10.746 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:15.762 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:16.683 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:16.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.799+0000 7fcaf8eaf640 10 monclient: get_monmap_and_config 2024-03-25T17:38:16.799 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.799+0000 7fcaf8eaf640 10 monclient: build_initial_monmap 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.799+0000 7fcaf8eaf640 10 monclient: monmap: 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:16.800287+0000 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:16.800287+0000 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:16.800 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:16.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.800+0000 7fcaf8eaf640 1 Processor -- start 2024-03-25T17:38:16.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.800+0000 7fcaf8eaf640 1 -- start start 2024-03-25T17:38:16.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.800+0000 7fcaf8eaf640 10 monclient: init 2024-03-25T17:38:16.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:16.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:16.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 10 monclient(hunting): picked mon.noname-a con 0x7fcaf40b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:16.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 10 monclient(hunting): picked mon.noname-b con 0x7fcaf40b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:16.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 10 monclient(hunting): picked mon.noname-c con 0x7fcaf414be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:16.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40af940 con 0x7fcaf40b4d60 2024-03-25T17:38:16.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40b0a70 con 0x7fcaf414be90 2024-03-25T17:38:16.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40b0d50 con 0x7fcaf40b1730 2024-03-25T17:38:16.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:16.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.801+0000 7fcaf8eaf640 10 monclient(hunting): authenticate will time out at 5850.064142s 2024-03-25T17:38:16.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.809+0000 7fcaf2d9d640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fcaf414be90 0x7fcaf414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:54448/0 (socket says 172.21.15.134:54448) 2024-03-25T17:38:16.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.809+0000 7fcaf2d9d640 1 -- 172.21.15.134:0/409784790 learned_addr learned my addr 172.21.15.134:0/409784790 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:16.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.826+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745647095 0 0) 0x7fcaf40af940 con 0x7fcaf40b4d60 2024-03-25T17:38:16.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.826+0000 7fcaf159a640 10 monclient(hunting): _negotiate 2024-03-25T17:38:16.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.826+0000 7fcaf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:16.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.826+0000 7fcaf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:16.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.826+0000 7fcaf159a640 10 monclient(hunting): my global_id is 4602 2024-03-25T17:38:16.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.826+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcadc003520 con 0x7fcaf40b4d60 2024-03-25T17:38:16.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.830+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198606048 0 0) 0x7fcaf40b0a70 con 0x7fcaf414be90 2024-03-25T17:38:16.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.830+0000 7fcaf159a640 10 monclient(hunting): _negotiate 2024-03-25T17:38:16.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.830+0000 7fcaf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:16.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.830+0000 7fcaf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:16.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.830+0000 7fcaf159a640 10 monclient(hunting): my global_id is 4633 2024-03-25T17:38:16.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.830+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf40af940 con 0x7fcaf414be90 2024-03-25T17:38:16.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.837+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283713335 0 0) 0x7fcaf40b0d50 con 0x7fcaf40b1730 2024-03-25T17:38:16.837 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.837+0000 7fcaf159a640 10 monclient(hunting): _negotiate 2024-03-25T17:38:16.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.837+0000 7fcaf159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:16.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.837+0000 7fcaf159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:16.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.837+0000 7fcaf159a640 10 monclient(hunting): my global_id is 34586 2024-03-25T17:38:16.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.837+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf40b0a70 con 0x7fcaf40b1730 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1439816303 0 0) 0x7fcaf40af940 con 0x7fcaf414be90 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 >> v1:172.21.15.134:6790/0 conn(0x7fcaf40b1730 legacy=0x7fcaf40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 >> v1:172.21.15.134:6789/0 conn(0x7fcaf40b4d60 legacy=0x7fcaf413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 10 monclient: _finish_hunting 0 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 1 monclient: found mon.noname-c 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:38:16.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf40b1030 con 0x7fcaf414be90 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 10 monclient: _finish_auth 0 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 10 monclient: _check_auth_tickets 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcaec0019f0 con 0x7fcaf414be90 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf8eaf640 5 monclient: authenticate success, global_id 4633 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf8eaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf159a640 10 monclient: dump: 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:16.844 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:16.845 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:16.845 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:16.845 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:16.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.842+0000 7fcaf8eaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:16.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcaec001b50 con 0x7fcaf414be90 2024-03-25T17:38:16.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:16.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf159a640 1 -- 172.21.15.134:0/409784790 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcaec001f60 con 0x7fcaf414be90 2024-03-25T17:38:16.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:16.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf159a640 10 monclient: dump: 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf8eaf640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:16.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf8eaf640 4 set_mon_vals no callback set 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf8eaf640 10 monclient: shutdown 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf8eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.850+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/409784790 >> v1:172.21.15.159:6789/0 conn(0x7fcaf414be90 legacy=0x7fcaf414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.851+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/409784790 shutdown_connections 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.851+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/409784790 >> 172.21.15.134:0/409784790 conn(0x7fcaf40617a0 msgr2=0x7fcaf4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.851+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/409784790 shutdown_connections 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.851+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/409784790 wait complete. 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 10 monclient: build_initial_monmap 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 10 monclient: monmap: 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:16.853 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 1 librados: starting msgr at 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 1 librados: starting objecter 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 1 Processor -- start 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 1 -- start start 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 1 librados: setting wanted keys 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 1 librados: calling monclient init 2024-03-25T17:38:16.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.852+0000 7fcaf8eaf640 10 monclient: init 2024-03-25T17:38:16.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 10 monclient(hunting): picked mon.noname-c con 0x7fcaf40b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 10 monclient(hunting): picked mon.noname-a con 0x7fcaf40b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 10 monclient(hunting): picked mon.noname-b con 0x7fcaf414be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40aabb0 con 0x7fcaf40b4d60 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40a95a0 con 0x7fcaf414be90 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf413b540 con 0x7fcaf40b1730 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:16.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.853+0000 7fcaf8eaf640 10 monclient(hunting): authenticate will time out at 5850.116685s 2024-03-25T17:38:16.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.863+0000 7fcaf1d9b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fcaf40b4d60 0x7fcaf41d7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:59606/0 (socket says 172.21.15.134:59606) 2024-03-25T17:38:16.864 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.863+0000 7fcaf1d9b640 1 -- 172.21.15.134:0/3174251697 learned_addr learned my addr 172.21.15.134:0/3174251697 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:16.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.865+0000 7fcae2ffd640 10 client.?.objecter ms_handle_connect 0x7fcaf40b4d60 2024-03-25T17:38:16.865 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.865+0000 7fcae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:16.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.874+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625939042 0 0) 0x7fcaf40aabb0 con 0x7fcaf40b4d60 2024-03-25T17:38:16.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.874+0000 7fcae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:16.874 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.874+0000 7fcae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:16.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.874+0000 7fcae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:16.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.874+0000 7fcae2ffd640 10 monclient(hunting): my global_id is 4605 2024-03-25T17:38:16.875 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.874+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcad4004eb0 con 0x7fcaf40b4d60 2024-03-25T17:38:16.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.875+0000 7fcae2ffd640 10 client.?.objecter ms_handle_connect 0x7fcaf414be90 2024-03-25T17:38:16.876 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.875+0000 7fcae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:16.878 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.878+0000 7fcae2ffd640 10 client.?.objecter ms_handle_connect 0x7fcaf40b1730 2024-03-25T17:38:16.878 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.878+0000 7fcae2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:16.892 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.892+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015509408 0 0) 0x7fcaf40a95a0 con 0x7fcaf414be90 2024-03-25T17:38:16.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.892+0000 7fcae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:16.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.892+0000 7fcae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:16.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.892+0000 7fcae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:16.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.892+0000 7fcae2ffd640 10 monclient(hunting): my global_id is 4636 2024-03-25T17:38:16.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.892+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf40aabb0 con 0x7fcaf414be90 2024-03-25T17:38:16.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.894+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753877635 0 0) 0x7fcaf413b540 con 0x7fcaf40b1730 2024-03-25T17:38:16.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.894+0000 7fcae2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:16.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.894+0000 7fcae2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:16.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.894+0000 7fcae2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:16.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.894+0000 7fcae2ffd640 10 monclient(hunting): my global_id is 34589 2024-03-25T17:38:16.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.894+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaf40a95a0 con 0x7fcaf40b1730 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.902+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2132013726 0 0) 0x7fcad4004eb0 con 0x7fcaf40b4d60 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.902+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcaf413b540 con 0x7fcaf40b4d60 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.902+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcae8001a10 con 0x7fcaf40b4d60 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.902+0000 7fcae2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.902+0000 7fcae2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.902+0000 7fcae2ffd640 10 monclient(hunting): dump: 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:16.903 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:16.904 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.902+0000 7fcae2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.914+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 756247652 0 0) 0x7fcaf40aabb0 con 0x7fcaf414be90 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.914+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcad4004eb0 con 0x7fcaf414be90 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.914+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcaec001e60 con 0x7fcaf414be90 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.914+0000 7fcae2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.914+0000 7fcae2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.914+0000 7fcae2ffd640 10 monclient(hunting): dump: 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:16.915 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:16.916 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:16.916 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:16.916 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:16.916 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:16.916 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:16.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3780833268 0 0) 0x7fcaf413b540 con 0x7fcaf40b4d60 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 >> v1:172.21.15.134:6790/0 conn(0x7fcaf40b1730 legacy=0x7fcaf4137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 >> v1:172.21.15.159:6789/0 conn(0x7fcaf414be90 legacy=0x7fcaf41d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 1 monclient: found mon.a 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf413b6d0 con 0x7fcaf40b4d60 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcaf8eaf640 5 monclient: authenticate success, global_id 4605 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcaf8eaf640 10 monclient: _renew_subs 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcaf8eaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.917+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf4138680 con 0x7fcaf40b4d60 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.918+0000 7fcaf8eaf640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:16.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.918+0000 7fcaf8eaf640 10 monclient: _renew_subs 2024-03-25T17:38:16.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.918+0000 7fcaf8eaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:16.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.918+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf4138d60 con 0x7fcaf40b4d60 2024-03-25T17:38:16.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.918+0000 7fcaf8eaf640 1 librados: init done 2024-03-25T17:38:16.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.919+0000 7fcaf8eaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:16.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.919+0000 7fcaf8eaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:16.920 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.919+0000 7fcaf8eaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:16.921 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:16.919+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf40afbd0 con 0x7fcaf40b4d60 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.203+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcae8001c70 con 0x7fcaf40b4d60 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.203+0000 7fcae2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.203+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fcae8003a60 con 0x7fcaf40b4d60 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.203+0000 7fcae2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.203+0000 7fcae2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.203+0000 7fcae2ffd640 10 monclient: dump: 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:17.204 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:17.205 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.203+0000 7fcaf0d99640 4 set_mon_vals no callback set 2024-03-25T17:38:17.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.204+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fcae8003c80 con 0x7fcaf40b4d60 2024-03-25T17:38:17.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.205+0000 7fcaf259c640 10 monclient: get_auth_request con 0x7fcad406df70 auth_method 0 2024-03-25T17:38:17.206 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.205+0000 7fcae2ffd640 10 client.4605.objecter ms_handle_connect 0x7fcad406df70 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fcae8054810 con 0x7fcaf40b4d60 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 10 client.4605.objecter ms_dispatch 0x7fcaf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 3 client.4605.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 3 client.4605.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcae8050320 con 0x7fcaf40b4d60 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:17.213 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.212+0000 7fcae2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:17.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.456+0000 7fcaf8eaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:17.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.456+0000 7fcaf8eaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:17.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.456+0000 7fcaf8eaf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:17.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.456+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaf40a9730 con 0x7fcaf40b4d60 2024-03-25T17:38:17.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.538+0000 7fcae2ffd640 1 -- 172.21.15.134:0/3174251697 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fcae8052250 con 0x7fcaf40b4d60 2024-03-25T17:38:17.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.538+0000 7fcae2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:17.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.538+0000 7fcae2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:17.539 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:17.539 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:17.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 10 librados: watch_flush enter 2024-03-25T17:38:17.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 10 librados: watch_flush exit 2024-03-25T17:38:17.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-25T17:38:17.542 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 10 client.4605.objecter successfully canceled tick 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 >> v1:172.21.15.134:6800/610837766 conn(0x7fcad406df70 legacy=0x7fcad4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 10 monclient: shutdown 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 >> v1:172.21.15.134:6789/0 conn(0x7fcaf40b4d60 legacy=0x7fcaf41d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 shutdown_connections 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 >> 172.21.15.134:0/3174251697 conn(0x7fcaf40617a0 msgr2=0x7fcaf413beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 shutdown_connections 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 1 -- 172.21.15.134:0/3174251697 wait complete. 2024-03-25T17:38:17.543 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:17.541+0000 7fcaf8eaf640 1 librados: shutdown 2024-03-25T17:38:17.553 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:24.360 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:24.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.476+0000 7f8df7767640 10 monclient: get_monmap_and_config 2024-03-25T17:38:24.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.476+0000 7f8df7767640 10 monclient: build_initial_monmap 2024-03-25T17:38:24.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.476+0000 7f8df7767640 10 monclient: monmap: 2024-03-25T17:38:24.477 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:24.477 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:24.477525+0000 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:24.477525+0000 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:24.478 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.477+0000 7f8df7767640 1 Processor -- start 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.477+0000 7f8df7767640 1 -- start start 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.477+0000 7f8df7767640 10 monclient: init 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 10 monclient(hunting): picked mon.noname-b con 0x7f8df00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 10 monclient(hunting): picked mon.noname-a con 0x7f8df013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:24.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 10 monclient(hunting): picked mon.noname-c con 0x7f8df01430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:24.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df0142520 con 0x7f8df013f710 2024-03-25T17:38:24.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00b28a0 con 0x7f8df01430c0 2024-03-25T17:38:24.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00b2a30 con 0x7f8df00a98f0 2024-03-25T17:38:24.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:24.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.478+0000 7f8df7767640 10 monclient(hunting): authenticate will time out at 5857.741104s 2024-03-25T17:38:24.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.485+0000 7f8df5503640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f8df00a98f0 0x7f8df00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:57292/0 (socket says 172.21.15.134:57292) 2024-03-25T17:38:24.486 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.485+0000 7f8df5503640 1 -- 172.21.15.134:0/2166855663 learned_addr learned my addr 172.21.15.134:0/2166855663 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:24.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.495+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339864722 0 0) 0x7f8df0142520 con 0x7f8df013f710 2024-03-25T17:38:24.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.495+0000 7f8de7fff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:24.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.495+0000 7f8de7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:24.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.495+0000 7f8de7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:24.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.495+0000 7f8de7fff640 10 monclient(hunting): my global_id is 4608 2024-03-25T17:38:24.496 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.495+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ddc003520 con 0x7f8df013f710 2024-03-25T17:38:24.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1863684649 0 0) 0x7f8ddc003520 con 0x7f8df013f710 2024-03-25T17:38:24.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 >> v1:172.21.15.134:6790/0 conn(0x7f8df00a98f0 legacy=0x7f8df00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:24.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 >> v1:172.21.15.159:6789/0 conn(0x7f8df01430c0 legacy=0x7f8df014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:24.504 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 10 monclient: _finish_hunting 0 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 1 monclient: found mon.noname-a 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df00b44c0 con 0x7f8df013f710 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 10 monclient: _finish_auth 0 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 10 monclient: _check_auth_tickets 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8de0001690 con 0x7f8df013f710 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8df7767640 5 monclient: authenticate success, global_id 4608 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8df7767640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:24.505 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8de7fff640 10 monclient: dump: 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:24.506 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.503+0000 7f8df7767640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:24.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8de00017f0 con 0x7f8df013f710 2024-03-25T17:38:24.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8de7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:24.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8de7fff640 1 -- 172.21.15.134:0/2166855663 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8de0001c00 con 0x7f8df013f710 2024-03-25T17:38:24.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8df7767640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:24.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8df7767640 4 set_mon_vals no callback set 2024-03-25T17:38:24.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8de7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8de7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.517+0000 7f8de7fff640 10 monclient: dump: 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:24.519 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:24.520 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.518+0000 7f8df7767640 10 monclient: shutdown 2024-03-25T17:38:24.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.518+0000 7f8df7767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:24.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.518+0000 7f8df7767640 1 -- 172.21.15.134:0/2166855663 >> v1:172.21.15.134:6789/0 conn(0x7f8df013f710 legacy=0x7f8df01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:24.522 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.518+0000 7f8df7767640 1 -- 172.21.15.134:0/2166855663 shutdown_connections 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.518+0000 7f8df7767640 1 -- 172.21.15.134:0/2166855663 >> 172.21.15.134:0/2166855663 conn(0x7f8df00b7ea0 msgr2=0x7f8df00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.518+0000 7f8df7767640 1 -- 172.21.15.134:0/2166855663 shutdown_connections 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.518+0000 7f8df7767640 1 -- 172.21.15.134:0/2166855663 wait complete. 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.519+0000 7f8df7767640 10 monclient: build_initial_monmap 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.519+0000 7f8df7767640 10 monclient: monmap: 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:24.523 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 librados: starting msgr at 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 librados: starting objecter 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 Processor -- start 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 -- start start 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 librados: setting wanted keys 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 librados: calling monclient init 2024-03-25T17:38:24.524 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient: init 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient(hunting): picked mon.noname-b con 0x7f8df00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient(hunting): picked mon.noname-a con 0x7f8df013f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient(hunting): picked mon.noname-c con 0x7f8df01430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df0138980 con 0x7f8df013f710 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df0138640 con 0x7f8df00a98f0 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df00b1700 con 0x7f8df01430c0 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:24.525 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.520+0000 7f8df7767640 10 monclient(hunting): authenticate will time out at 5857.783744s 2024-03-25T17:38:24.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.536+0000 7f8df5503640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f8df00a98f0 0x7f8df0137960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:56730/0 (socket says 172.21.15.134:56730) 2024-03-25T17:38:24.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.536+0000 7f8df5503640 1 -- 172.21.15.134:0/4037348065 learned_addr learned my addr 172.21.15.134:0/4037348065 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:24.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.540+0000 7f8de5ffb640 10 client.?.objecter ms_handle_connect 0x7f8df00a98f0 2024-03-25T17:38:24.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.540+0000 7f8de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:24.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.544+0000 7f8de5ffb640 10 client.?.objecter ms_handle_connect 0x7f8df01430c0 2024-03-25T17:38:24.545 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.544+0000 7f8de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:24.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.546+0000 7f8de5ffb640 10 client.?.objecter ms_handle_connect 0x7f8df013f710 2024-03-25T17:38:24.547 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.546+0000 7f8de5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:24.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.549+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111462444 0 0) 0x7f8df0138640 con 0x7f8df00a98f0 2024-03-25T17:38:24.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.549+0000 7f8de5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:24.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.549+0000 7f8de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:24.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.549+0000 7f8de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:24.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.549+0000 7f8de5ffb640 10 monclient(hunting): my global_id is 4642 2024-03-25T17:38:24.550 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.549+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8dd4004eb0 con 0x7f8df00a98f0 2024-03-25T17:38:24.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.559+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345351414 0 0) 0x7f8df00b1700 con 0x7f8df01430c0 2024-03-25T17:38:24.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.559+0000 7f8de5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:24.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.559+0000 7f8de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:24.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.559+0000 7f8de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:24.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.559+0000 7f8de5ffb640 10 monclient(hunting): my global_id is 34595 2024-03-25T17:38:24.560 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.559+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df0138640 con 0x7f8df01430c0 2024-03-25T17:38:24.564 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.564+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630906393 0 0) 0x7f8df0138980 con 0x7f8df013f710 2024-03-25T17:38:24.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.564+0000 7f8de5ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:24.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.564+0000 7f8de5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:24.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.564+0000 7f8de5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:24.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.564+0000 7f8de5ffb640 10 monclient(hunting): my global_id is 4611 2024-03-25T17:38:24.565 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.564+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8df00b1700 con 0x7f8df013f710 2024-03-25T17:38:24.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.582+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 322189902 0 0) 0x7f8dd4004eb0 con 0x7f8df00a98f0 2024-03-25T17:38:24.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.582+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8df0138980 con 0x7f8df00a98f0 2024-03-25T17:38:24.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.582+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8dec0018b0 con 0x7f8df00a98f0 2024-03-25T17:38:24.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.582+0000 7f8de5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:24.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.582+0000 7f8de5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:24.583 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.582+0000 7f8de5ffb640 10 monclient(hunting): dump: 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.584 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.582+0000 7f8de5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:24.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.583+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3399784301 0 0) 0x7f8df0138640 con 0x7f8df01430c0 2024-03-25T17:38:24.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.583+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8dd4004eb0 con 0x7f8df01430c0 2024-03-25T17:38:24.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.583+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8de8001b10 con 0x7f8df01430c0 2024-03-25T17:38:24.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.583+0000 7f8de5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:24.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.583+0000 7f8de5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:24.585 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.583+0000 7f8de5ffb640 10 monclient(hunting): dump: 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:24.586 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.596+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359841726 0 0) 0x7f8df00b1700 con 0x7f8df013f710 2024-03-25T17:38:24.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.596+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8df0138640 con 0x7f8df013f710 2024-03-25T17:38:24.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.596+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8de0001b00 con 0x7f8df013f710 2024-03-25T17:38:24.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.596+0000 7f8de5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:24.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.596+0000 7f8de5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.596+0000 7f8de5ffb640 10 monclient(hunting): dump: 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:24.598 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 272719941 0 0) 0x7f8dd4004eb0 con 0x7f8df01430c0 2024-03-25T17:38:24.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 >> v1:172.21.15.159:6789/0 conn(0x7f8df00a98f0 legacy=0x7f8df0137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:24.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 >> v1:172.21.15.134:6789/0 conn(0x7f8df013f710 legacy=0x7f8df01da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:24.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:38:24.602 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 1 monclient: found mon.c 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df00b1290 con 0x7f8df01430c0 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 10 monclient: _finish_auth 0 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8de5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 5 monclient: authenticate success, global_id 34595 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 10 monclient: _renew_subs 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df01e2eb0 con 0x7f8df01430c0 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 10 client.34595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:24.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 10 monclient: _renew_subs 2024-03-25T17:38:24.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:38:24.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df01e3220 con 0x7f8df01430c0 2024-03-25T17:38:24.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.601+0000 7f8df7767640 1 librados: init done 2024-03-25T17:38:24.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.602+0000 7f8df7767640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:24.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.602+0000 7f8df7767640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:24.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.602+0000 7f8df7767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:38:24.605 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.602+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df00b14e0 con 0x7f8df01430c0 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.892+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8de8002000 con 0x7f8df01430c0 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.892+0000 7f8de5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.892+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8de8003970 con 0x7f8df01430c0 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.892+0000 7f8de5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.892+0000 7f8de5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.892+0000 7f8de5ffb640 10 monclient: dump: 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.893 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:24.894 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:24.894 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:24.894 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:24.894 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:24.894 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:24.894 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:24.894 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.892+0000 7f8de77fe640 4 set_mon_vals no callback set 2024-03-25T17:38:24.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.893+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f8de8019940 con 0x7f8df01430c0 2024-03-25T17:38:24.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.893+0000 7f8df5503640 10 monclient: get_auth_request con 0x7f8dd406e160 auth_method 0 2024-03-25T17:38:24.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.893+0000 7f8de5ffb640 10 client.34595.objecter ms_handle_connect 0x7f8dd406e160 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f8de8053600 con 0x7f8df01430c0 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 10 client.34595.objecter ms_dispatch 0x7f8df0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 3 client.34595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 3 client.34595.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 20 client.34595.objecter dump_active .. 0 homeless 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8de804ef40 con 0x7f8df01430c0 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:24.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:24.902+0000 7f8de5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:25.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.189+0000 7f8df7767640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:25.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.189+0000 7f8df7767640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:25.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.189+0000 7f8df7767640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:38:25.190 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.189+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8df00627b0 con 0x7f8df01430c0 2024-03-25T17:38:25.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.216+0000 7f8de5ffb640 1 -- 172.21.15.134:0/4037348065 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f8de8051040 con 0x7f8df01430c0 2024-03-25T17:38:25.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.216+0000 7f8de5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:25.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.216+0000 7f8de5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:25.217 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:25.217 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:25.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.218+0000 7f8df7767640 10 librados: watch_flush enter 2024-03-25T17:38:25.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.218+0000 7f8df7767640 10 librados: watch_flush exit 2024-03-25T17:38:25.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.218+0000 7f8df7767640 20 client.34595.objecter shutdown clearing up homeless session... 2024-03-25T17:38:25.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.218+0000 7f8df7767640 10 client.34595.objecter successfully canceled tick 2024-03-25T17:38:25.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 >> v1:172.21.15.134:6800/610837766 conn(0x7f8dd406e160 legacy=0x7f8dd4070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:25.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 10 monclient: shutdown 2024-03-25T17:38:25.220 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:25.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 >> v1:172.21.15.134:6790/0 conn(0x7f8df01430c0 legacy=0x7f8df01db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:25.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 shutdown_connections 2024-03-25T17:38:25.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 >> 172.21.15.134:0/4037348065 conn(0x7f8df00b7ea0 msgr2=0x7f8df0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:25.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 shutdown_connections 2024-03-25T17:38:25.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 1 -- 172.21.15.134:0/4037348065 wait complete. 2024-03-25T17:38:25.221 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:25.219+0000 7f8df7767640 1 librados: shutdown 2024-03-25T17:38:25.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:30.751 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:35.768 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.796 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:35.812 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.826 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:35.842 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.858 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:35.887 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.863 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:35.880 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.867 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:35.896 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.873 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:35.889 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.895 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:35.924 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:30.916 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:35.932 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:32.038 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.154+0000 7f8594093640 10 monclient: get_monmap_and_config 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.154+0000 7f8594093640 10 monclient: build_initial_monmap 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.154+0000 7f8594093640 10 monclient: monmap: 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:32.155493+0000 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:32.155493+0000 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:32.155 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:32.156 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:32.156 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:32.156 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.155+0000 7f8594093640 1 Processor -- start 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.155+0000 7f8594093640 1 -- start start 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.155+0000 7f8594093640 10 monclient: init 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 10 monclient(hunting): picked mon.noname-a con 0x7f858c06adf0 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 10 monclient(hunting): picked mon.noname-b con 0x7f858c0aee10 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 10 monclient(hunting): picked mon.noname-c con 0x7f858c140860 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c0af690 con 0x7f858c06adf0 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c0b0c70 con 0x7f858c140860 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c0b0f50 con 0x7f858c0aee10 2024-03-25T17:38:32.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:32.158 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.156+0000 7f8594093640 10 monclient(hunting): authenticate will time out at 5865.419032s 2024-03-25T17:38:32.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.167+0000 7f8591e2f640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f858c06adf0 0x7f858c13d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:48560/0 (socket says 172.21.15.134:48560) 2024-03-25T17:38:32.167 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.167+0000 7f8591e2f640 1 -- 172.21.15.134:0/3429268508 learned_addr learned my addr 172.21.15.134:0/3429268508 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:32.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.174+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902861376 0 0) 0x7f858c0af690 con 0x7f858c06adf0 2024-03-25T17:38:32.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.174+0000 7f8590e2d640 10 monclient(hunting): _negotiate 2024-03-25T17:38:32.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.174+0000 7f8590e2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:32.174 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.174+0000 7f8590e2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:32.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.174+0000 7f8590e2d640 10 monclient(hunting): my global_id is 4614 2024-03-25T17:38:32.175 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.174+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8578003520 con 0x7f858c06adf0 2024-03-25T17:38:32.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097601237 0 0) 0x7f8578003520 con 0x7f858c06adf0 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 >> v1:172.21.15.134:6790/0 conn(0x7f858c0aee10 legacy=0x7f858c140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 >> v1:172.21.15.159:6789/0 conn(0x7f858c140860 legacy=0x7f858c14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 10 monclient: _finish_hunting 0 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 1 monclient: found mon.noname-a 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c13dc40 con 0x7f858c06adf0 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 10 monclient: _finish_auth 0 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 10 monclient: _check_auth_tickets 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8594093640 5 monclient: authenticate success, global_id 4614 2024-03-25T17:38:32.185 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8594093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8588001690 con 0x7f858c06adf0 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8590e2d640 10 monclient: dump: 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.186 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.187 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:32.187 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:32.187 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:32.187 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:32.187 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:32.187 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:32.187 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.184+0000 7f8594093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:32.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8588001920 con 0x7f858c06adf0 2024-03-25T17:38:32.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8590e2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:32.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8590e2d640 1 -- 172.21.15.134:0/3429268508 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8588001c30 con 0x7f858c06adf0 2024-03-25T17:38:32.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8594093640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:32.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8594093640 4 set_mon_vals no callback set 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8590e2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8590e2d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8590e2d640 10 monclient: dump: 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:32.192 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8594093640 10 monclient: shutdown 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8594093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8594093640 1 -- 172.21.15.134:0/3429268508 >> v1:172.21.15.134:6789/0 conn(0x7f858c06adf0 legacy=0x7f858c13d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8594093640 1 -- 172.21.15.134:0/3429268508 shutdown_connections 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.191+0000 7f8594093640 1 -- 172.21.15.134:0/3429268508 >> 172.21.15.134:0/3429268508 conn(0x7f858c0b4a80 msgr2=0x7f858c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.192+0000 7f8594093640 1 -- 172.21.15.134:0/3429268508 shutdown_connections 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.192+0000 7f8594093640 1 -- 172.21.15.134:0/3429268508 wait complete. 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.192+0000 7f8594093640 10 monclient: build_initial_monmap 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.192+0000 7f8594093640 10 monclient: monmap: 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:32.194 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.193+0000 7f8594093640 1 librados: starting msgr at 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.193+0000 7f8594093640 1 librados: starting objecter 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.193+0000 7f8594093640 1 Processor -- start 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.193+0000 7f8594093640 1 -- start start 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.193+0000 7f8594093640 1 librados: setting wanted keys 2024-03-25T17:38:32.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.193+0000 7f8594093640 1 librados: calling monclient init 2024-03-25T17:38:32.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.193+0000 7f8594093640 10 monclient: init 2024-03-25T17:38:32.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:32.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 10 monclient(hunting): picked mon.noname-b con 0x7f858c06adf0 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 10 monclient(hunting): picked mon.noname-a con 0x7f858c0aee10 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 10 monclient(hunting): picked mon.noname-c con 0x7f858c137b40 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c1d7bd0 con 0x7f858c0aee10 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c134660 con 0x7f858c06adf0 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c1d8770 con 0x7f858c137b40 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:32.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.194+0000 7f8594093640 10 monclient(hunting): authenticate will time out at 5865.457104s 2024-03-25T17:38:32.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.210+0000 7f8592630640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f858c137b40 0x7f858c135ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:42208/0 (socket says 172.21.15.134:42208) 2024-03-25T17:38:32.211 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.210+0000 7f8592630640 1 -- 172.21.15.134:0/3417470314 learned_addr learned my addr 172.21.15.134:0/3417470314 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:32.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.214+0000 7f85827fc640 10 client.?.objecter ms_handle_connect 0x7f858c137b40 2024-03-25T17:38:32.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.214+0000 7f85827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:32.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.216+0000 7f85827fc640 10 client.?.objecter ms_handle_connect 0x7f858c06adf0 2024-03-25T17:38:32.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.216+0000 7f85827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:32.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.218+0000 7f85827fc640 10 client.?.objecter ms_handle_connect 0x7f858c0aee10 2024-03-25T17:38:32.219 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.218+0000 7f85827fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:32.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.227+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533724209 0 0) 0x7f858c1d7bd0 con 0x7f858c0aee10 2024-03-25T17:38:32.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:32.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:32.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:32.228 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): my global_id is 4617 2024-03-25T17:38:32.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8570004eb0 con 0x7f858c0aee10 2024-03-25T17:38:32.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013416017 0 0) 0x7f858c1d8770 con 0x7f858c137b40 2024-03-25T17:38:32.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:32.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:32.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:32.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 10 monclient(hunting): my global_id is 34601 2024-03-25T17:38:32.229 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.228+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f858c1d7bd0 con 0x7f858c137b40 2024-03-25T17:38:32.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.232+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520640726 0 0) 0x7f858c134660 con 0x7f858c06adf0 2024-03-25T17:38:32.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.233+0000 7f85827fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:32.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.233+0000 7f85827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:32.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.233+0000 7f85827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:32.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.233+0000 7f85827fc640 10 monclient(hunting): my global_id is 4648 2024-03-25T17:38:32.233 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.233+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f858c1d8770 con 0x7f858c06adf0 2024-03-25T17:38:32.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.244+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2601826943 0 0) 0x7f8570004eb0 con 0x7f858c0aee10 2024-03-25T17:38:32.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.244+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f858c134660 con 0x7f858c0aee10 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.244+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f857c0018c0 con 0x7f858c0aee10 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.244+0000 7f85827fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.244+0000 7f85827fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.244+0000 7f85827fc640 10 monclient(hunting): dump: 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:32.245 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.244+0000 7f85827fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2534632231 0 0) 0x7f858c134660 con 0x7f858c0aee10 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 >> v1:172.21.15.134:6790/0 conn(0x7f858c137b40 legacy=0x7f858c135ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 >> v1:172.21.15.159:6789/0 conn(0x7f858c06adf0 legacy=0x7f858c0b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 10 monclient: _finish_hunting 0 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 1 monclient: found mon.a 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:32.251 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c1d8900 con 0x7f858c0aee10 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 10 monclient: _finish_auth 0 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 10 monclient: _check_auth_tickets 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f85827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 5 monclient: authenticate success, global_id 4617 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 10 monclient: _renew_subs 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c0b3130 con 0x7f858c0aee10 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 10 monclient: _renew_subs 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c1365d0 con 0x7f858c0aee10 2024-03-25T17:38:32.252 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.250+0000 7f8594093640 1 librados: init done 2024-03-25T17:38:32.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.251+0000 7f8594093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:32.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.251+0000 7f8594093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:32.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.251+0000 7f8594093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:32.253 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.251+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c1344a0 con 0x7f858c0aee10 2024-03-25T17:38:32.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.461+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f857c0035f0 con 0x7f858c0aee10 2024-03-25T17:38:32.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.461+0000 7f85827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:32.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.461+0000 7f8583fff640 4 set_mon_vals no callback set 2024-03-25T17:38:32.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.502+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f857c003a00 con 0x7f858c0aee10 2024-03-25T17:38:32.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.502+0000 7f85827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:32.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.502+0000 7f85827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:32.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.502+0000 7f85827fc640 10 monclient: dump: 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:32.503 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:32.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.551+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f857c003be0 con 0x7f858c0aee10 2024-03-25T17:38:32.552 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.552+0000 7f8591e2f640 10 monclient: get_auth_request con 0x7f857006ddd0 auth_method 0 2024-03-25T17:38:32.553 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.552+0000 7f85827fc640 10 client.4617.objecter ms_handle_connect 0x7f857006ddd0 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f857c053620 con 0x7f858c0aee10 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 10 client.4617.objecter ms_dispatch 0x7f858c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f857c04f130 con 0x7f858c0aee10 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:32.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.558+0000 7f85827fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:32.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.810+0000 7f8594093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:32.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.810+0000 7f8594093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:32.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.810+0000 7f8594093640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:32.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.810+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f858c1d7d80 con 0x7f858c0aee10 2024-03-25T17:38:32.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.838+0000 7f85827fc640 1 -- 172.21.15.134:0/3417470314 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f858c1d7d80 con 0x7f858c0aee10 2024-03-25T17:38:32.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.838+0000 7f85827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:32.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.838+0000 7f85827fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:32.839 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:32.839 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 10 librados: watch_flush enter 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 10 librados: watch_flush exit 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 10 client.4617.objecter successfully canceled tick 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 >> v1:172.21.15.134:6800/610837766 conn(0x7f857006ddd0 legacy=0x7f8570070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 10 monclient: shutdown 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 >> v1:172.21.15.134:6789/0 conn(0x7f858c0aee10 legacy=0x7f858c1d8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 shutdown_connections 2024-03-25T17:38:32.842 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 >> 172.21.15.134:0/3417470314 conn(0x7f858c0b4a80 msgr2=0x7f858c1331e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:32.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 shutdown_connections 2024-03-25T17:38:32.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 1 -- 172.21.15.134:0/3417470314 wait complete. 2024-03-25T17:38:32.843 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:32.841+0000 7f8594093640 1 librados: shutdown 2024-03-25T17:38:32.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:39.659 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:39.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.772+0000 7f39b6344640 10 monclient: get_monmap_and_config 2024-03-25T17:38:39.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.772+0000 7f39b6344640 10 monclient: build_initial_monmap 2024-03-25T17:38:39.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.772+0000 7f39b6344640 10 monclient: monmap: 2024-03-25T17:38:39.773 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:39.773 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:39.773 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:39.773732+0000 2024-03-25T17:38:39.774 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:39.773732+0000 2024-03-25T17:38:39.774 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:39.774 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:39.774 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:39.774 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:39.774 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:39.774 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:39.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.773+0000 7f39b6344640 1 Processor -- start 2024-03-25T17:38:39.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.773+0000 7f39b6344640 1 -- start start 2024-03-25T17:38:39.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.773+0000 7f39b6344640 10 monclient: init 2024-03-25T17:38:39.775 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 10 monclient(hunting): picked mon.noname-b con 0x7f39b00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 10 monclient(hunting): picked mon.noname-c con 0x7f39b013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 10 monclient(hunting): picked mon.noname-a con 0x7f39b01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0142520 con 0x7f39b01430c0 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b28a0 con 0x7f39b013f710 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b2a30 con 0x7f39b00a98f0 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:39.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.774+0000 7f39b6344640 10 monclient(hunting): authenticate will time out at 5873.037496s 2024-03-25T17:38:39.780 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.780+0000 7f39b48e1640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f39b01430c0 0x7f39b014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:52296/0 (socket says 172.21.15.134:52296) 2024-03-25T17:38:39.781 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.780+0000 7f39b48e1640 1 -- 172.21.15.134:0/2338779544 learned_addr learned my addr 172.21.15.134:0/2338779544 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:39.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.788+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3324820102 0 0) 0x7f39b0142520 con 0x7f39b01430c0 2024-03-25T17:38:39.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.788+0000 7f39aeffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:39.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.788+0000 7f39aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:39.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.788+0000 7f39aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:39.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.788+0000 7f39aeffd640 10 monclient(hunting): my global_id is 4620 2024-03-25T17:38:39.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.788+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3998003520 con 0x7f39b01430c0 2024-03-25T17:38:39.789 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.789+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327056547 0 0) 0x7f39b00b2a30 con 0x7f39b00a98f0 2024-03-25T17:38:39.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.789+0000 7f39aeffd640 10 monclient(hunting): _negotiate 2024-03-25T17:38:39.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.789+0000 7f39aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:39.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.789+0000 7f39aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:39.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.789+0000 7f39aeffd640 10 monclient(hunting): my global_id is 34604 2024-03-25T17:38:39.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.789+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0142520 con 0x7f39b00a98f0 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1128617135 0 0) 0x7f3998003520 con 0x7f39b01430c0 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 >> v1:172.21.15.134:6790/0 conn(0x7f39b00a98f0 legacy=0x7f39b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 >> v1:172.21.15.159:6789/0 conn(0x7f39b013f710 legacy=0x7f39b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 10 monclient: _finish_hunting 0 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 1 monclient: found mon.noname-a 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b00b44c0 con 0x7f39b01430c0 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 10 monclient: _finish_auth 0 2024-03-25T17:38:39.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 10 monclient: _check_auth_tickets 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f39a8001a10 con 0x7f39b01430c0 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39b6344640 5 monclient: authenticate success, global_id 4620 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39b6344640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39aeffd640 10 monclient: dump: 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:39.803 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:39.804 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:39.804 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:39.804 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:39.804 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:39.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.801+0000 7f39b6344640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:39.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a8001b70 con 0x7f39b01430c0 2024-03-25T17:38:39.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:39.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39aeffd640 1 -- 172.21.15.134:0/2338779544 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f39a8001f80 con 0x7f39b01430c0 2024-03-25T17:38:39.809 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39b6344640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39b6344640 4 set_mon_vals no callback set 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39aeffd640 10 monclient: dump: 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:39.810 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:39.811 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:39.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39b6344640 10 monclient: shutdown 2024-03-25T17:38:39.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39b6344640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:39.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39b6344640 1 -- 172.21.15.134:0/2338779544 >> v1:172.21.15.134:6789/0 conn(0x7f39b01430c0 legacy=0x7f39b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39b6344640 1 -- 172.21.15.134:0/2338779544 shutdown_connections 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.809+0000 7f39b6344640 1 -- 172.21.15.134:0/2338779544 >> 172.21.15.134:0/2338779544 conn(0x7f39b00b7ea0 msgr2=0x7f39b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.810+0000 7f39b6344640 1 -- 172.21.15.134:0/2338779544 shutdown_connections 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.810+0000 7f39b6344640 1 -- 172.21.15.134:0/2338779544 wait complete. 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.810+0000 7f39b6344640 10 monclient: build_initial_monmap 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.810+0000 7f39b6344640 10 monclient: monmap: 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:39.813 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.810+0000 7f39b6344640 1 librados: starting msgr at 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.810+0000 7f39b6344640 1 librados: starting objecter 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 1 Processor -- start 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 1 -- start start 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 1 librados: setting wanted keys 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 1 librados: calling monclient init 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 10 monclient: init 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 10 monclient(hunting): picked mon.noname-b con 0x7f39b00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 10 monclient(hunting): picked mon.noname-c con 0x7f39b013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.811+0000 7f39b6344640 10 monclient(hunting): picked mon.noname-a con 0x7f39b01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:39.814 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.812+0000 7f39b6344640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b0136a50 con 0x7f39b01430c0 2024-03-25T17:38:39.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.812+0000 7f39b6344640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b79c0 con 0x7f39b00a98f0 2024-03-25T17:38:39.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.812+0000 7f39b6344640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b00b7ca0 con 0x7f39b013f710 2024-03-25T17:38:39.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.812+0000 7f39b6344640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:39.815 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.812+0000 7f39b6344640 10 monclient(hunting): authenticate will time out at 5873.074816s 2024-03-25T17:38:39.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.822+0000 7f39af7fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f39b013f710 0x7f39b0135350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:36838/0 (socket says 172.21.15.134:36838) 2024-03-25T17:38:39.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.822+0000 7f39af7fe640 1 -- 172.21.15.134:0/3788762326 learned_addr learned my addr 172.21.15.134:0/3788762326 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:39.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.824+0000 7f39acff9640 10 client.?.objecter ms_handle_connect 0x7f39b01430c0 2024-03-25T17:38:39.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.824+0000 7f39acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:39.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.827+0000 7f39acff9640 10 client.?.objecter ms_handle_connect 0x7f39b013f710 2024-03-25T17:38:39.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.827+0000 7f39acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:39.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.829+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857880225 0 0) 0x7f39b0136a50 con 0x7f39b01430c0 2024-03-25T17:38:39.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.829+0000 7f39acff9640 10 monclient(hunting): _negotiate 2024-03-25T17:38:39.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.829+0000 7f39acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:39.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.829+0000 7f39acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:39.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.829+0000 7f39acff9640 10 monclient(hunting): my global_id is 4623 2024-03-25T17:38:39.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.829+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f399c004eb0 con 0x7f39b01430c0 2024-03-25T17:38:39.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.836+0000 7f39acff9640 10 client.?.objecter ms_handle_connect 0x7f39b00a98f0 2024-03-25T17:38:39.836 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.836+0000 7f39acff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:39.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.838+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952649038 0 0) 0x7f39b00b7ca0 con 0x7f39b013f710 2024-03-25T17:38:39.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.838+0000 7f39acff9640 10 monclient(hunting): _negotiate 2024-03-25T17:38:39.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.838+0000 7f39acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:39.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.838+0000 7f39acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:39.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.838+0000 7f39acff9640 10 monclient(hunting): my global_id is 34607 2024-03-25T17:38:39.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.838+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b0136a50 con 0x7f39b013f710 2024-03-25T17:38:39.844 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.844+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285144963 0 0) 0x7f39b00b79c0 con 0x7f39b00a98f0 2024-03-25T17:38:39.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.844+0000 7f39acff9640 10 monclient(hunting): _negotiate 2024-03-25T17:38:39.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.844+0000 7f39acff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:39.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.844+0000 7f39acff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:39.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.844+0000 7f39acff9640 10 monclient(hunting): my global_id is 4654 2024-03-25T17:38:39.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.844+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39b00b7ca0 con 0x7f39b00a98f0 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.845+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1790736469 0 0) 0x7f399c004eb0 con 0x7f39b01430c0 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.845+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39b00b79c0 con 0x7f39b01430c0 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.845+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f39a8001e80 con 0x7f39b01430c0 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.845+0000 7f39acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.845+0000 7f39acff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.845+0000 7f39acff9640 10 monclient(hunting): dump: 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.846 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.847 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:39.847 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:39.847 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:39.847 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:39.847 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:39.847 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:39.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.845+0000 7f39acff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:39.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.852+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707801234 0 0) 0x7f39b0136a50 con 0x7f39b013f710 2024-03-25T17:38:39.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.852+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f399c004eb0 con 0x7f39b013f710 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.852+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f39a40019b0 con 0x7f39b013f710 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.852+0000 7f39acff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.852+0000 7f39acff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.852+0000 7f39acff9640 10 monclient(hunting): dump: 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:39.853 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 544687511 0 0) 0x7f39b00b79c0 con 0x7f39b01430c0 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 >> v1:172.21.15.134:6790/0 conn(0x7f39b013f710 legacy=0x7f39b0135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 >> v1:172.21.15.159:6789/0 conn(0x7f39b00a98f0 legacy=0x7f39b0137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 10 monclient: _finish_hunting 0 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 1 monclient: found mon.a 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b0136ca0 con 0x7f39b01430c0 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 10 monclient: _finish_auth 0 2024-03-25T17:38:39.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 10 monclient: _check_auth_tickets 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 5 monclient: authenticate success, global_id 4623 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 10 monclient: _renew_subs 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0136f80 con 0x7f39b01430c0 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 10 client.4623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 10 monclient: _renew_subs 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b01360e0 con 0x7f39b01430c0 2024-03-25T17:38:39.856 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.854+0000 7f39b6344640 1 librados: init done 2024-03-25T17:38:39.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.855+0000 7f39b6344640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:39.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.855+0000 7f39b6344640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:39.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.855+0000 7f39b6344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:39.857 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:39.855+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b00627b0 con 0x7f39b01430c0 2024-03-25T17:38:40.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.154+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39a8002020 con 0x7f39b01430c0 2024-03-25T17:38:40.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.154+0000 7f39acff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:40.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.154+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f39a8004fb0 con 0x7f39b01430c0 2024-03-25T17:38:40.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.154+0000 7f39acff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:40.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.154+0000 7f39acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:40.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.154+0000 7f39acff9640 10 monclient: dump: 2024-03-25T17:38:40.155 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:40.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.154+0000 7f39ae7fc640 4 set_mon_vals no callback set 2024-03-25T17:38:40.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.155+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f39a801af80 con 0x7f39b01430c0 2024-03-25T17:38:40.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.156+0000 7f39affff640 10 monclient: get_auth_request con 0x7f399c06df70 auth_method 0 2024-03-25T17:38:40.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.156+0000 7f39acff9640 10 client.4623.objecter ms_handle_connect 0x7f399c06df70 2024-03-25T17:38:40.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f39a8054ac0 con 0x7f39b01430c0 2024-03-25T17:38:40.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 10 client.4623.objecter ms_dispatch 0x7f39b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:40.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 3 client.4623.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:40.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 3 client.4623.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:40.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 20 client.4623.objecter dump_active .. 0 homeless 2024-03-25T17:38:40.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39a80505d0 con 0x7f39b01430c0 2024-03-25T17:38:40.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:40.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.160+0000 7f39acff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:40.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.411+0000 7f39b6344640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:40.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.411+0000 7f39b6344640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:40.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.411+0000 7f39b6344640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:40.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.411+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39b0135070 con 0x7f39b01430c0 2024-03-25T17:38:40.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.436+0000 7f39acff9640 1 -- 172.21.15.134:0/3788762326 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f39a8052500 con 0x7f39b01430c0 2024-03-25T17:38:40.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.436+0000 7f39acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:40.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.436+0000 7f39acff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:40.437 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:40.437 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:40.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.438+0000 7f39b6344640 10 librados: watch_flush enter 2024-03-25T17:38:40.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.438+0000 7f39b6344640 10 librados: watch_flush exit 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 20 client.4623.objecter shutdown clearing up homeless session... 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 10 client.4623.objecter successfully canceled tick 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 >> v1:172.21.15.134:6800/610837766 conn(0x7f399c06df70 legacy=0x7f399c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 10 monclient: shutdown 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 >> v1:172.21.15.134:6789/0 conn(0x7f39b01430c0 legacy=0x7f39b01e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 shutdown_connections 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.439+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 >> 172.21.15.134:0/3788762326 conn(0x7f39b00b7ea0 msgr2=0x7f39b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:40.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.440+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 shutdown_connections 2024-03-25T17:38:40.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.440+0000 7f39b6344640 1 -- 172.21.15.134:0/3788762326 wait complete. 2024-03-25T17:38:40.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:40.440+0000 7f39b6344640 1 librados: shutdown 2024-03-25T17:38:40.452 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:47.259 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:47.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.376+0000 7fbf59363640 10 monclient: get_monmap_and_config 2024-03-25T17:38:47.377 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.377+0000 7fbf59363640 10 monclient: build_initial_monmap 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.377+0000 7fbf59363640 10 monclient: monmap: 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:47.378011+0000 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:47.378011+0000 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:47.378 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:47.379 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:47.379 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.377+0000 7fbf59363640 1 Processor -- start 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.377+0000 7fbf59363640 1 -- start start 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.377+0000 7fbf59363640 10 monclient: init 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 10 monclient(hunting): picked mon.noname-c con 0x7fbf540a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:47.380 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 10 monclient(hunting): picked mon.noname-b con 0x7fbf541430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:47.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf54142520 con 0x7fbf5413f710 2024-03-25T17:38:47.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540b28a0 con 0x7fbf540a98f0 2024-03-25T17:38:47.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540b2a30 con 0x7fbf541430c0 2024-03-25T17:38:47.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:47.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.378+0000 7fbf59363640 10 monclient(hunting): authenticate will time out at 5880.641456s 2024-03-25T17:38:47.381 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.381+0000 7fbf4a7fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbf5413f710 0x7fbf541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:57384/0 (socket says 172.21.15.134:57384) 2024-03-25T17:38:47.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.381+0000 7fbf4a7fc640 1 -- 172.21.15.134:0/503028088 learned_addr learned my addr 172.21.15.134:0/503028088 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:47.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.393+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787533622 0 0) 0x7fbf540b2a30 con 0x7fbf541430c0 2024-03-25T17:38:47.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.393+0000 7fbf527fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:47.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.393+0000 7fbf527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:47.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.393+0000 7fbf527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:47.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.393+0000 7fbf527fc640 10 monclient(hunting): my global_id is 34610 2024-03-25T17:38:47.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.393+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf40003520 con 0x7fbf541430c0 2024-03-25T17:38:47.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.395+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909059415 0 0) 0x7fbf54142520 con 0x7fbf5413f710 2024-03-25T17:38:47.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.395+0000 7fbf527fc640 10 monclient(hunting): _negotiate 2024-03-25T17:38:47.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.395+0000 7fbf527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:47.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.395+0000 7fbf527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:47.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.395+0000 7fbf527fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T17:38:47.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.395+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf540b2a30 con 0x7fbf5413f710 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3049908190 0 0) 0x7fbf540b2a30 con 0x7fbf5413f710 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 >> v1:172.21.15.134:6790/0 conn(0x7fbf541430c0 legacy=0x7fbf5414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 >> v1:172.21.15.159:6789/0 conn(0x7fbf540a98f0 legacy=0x7fbf540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 10 monclient: _finish_hunting 0 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 1 monclient: found mon.noname-a 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540b44c0 con 0x7fbf5413f710 2024-03-25T17:38:47.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 10 monclient: _finish_auth 0 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 10 monclient: _check_auth_tickets 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbf3c001670 con 0x7fbf5413f710 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf527fc640 10 monclient: dump: 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:47.408 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:47.409 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:47.409 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:47.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf59363640 5 monclient: authenticate success, global_id 4626 2024-03-25T17:38:47.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.406+0000 7fbf59363640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:47.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf3c0017d0 con 0x7fbf5413f710 2024-03-25T17:38:47.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:47.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf527fc640 1 -- 172.21.15.134:0/503028088 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbf3c001ba0 con 0x7fbf5413f710 2024-03-25T17:38:47.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf59363640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:47.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf59363640 4 set_mon_vals no callback set 2024-03-25T17:38:47.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:47.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.432+0000 7fbf527fc640 10 monclient: dump: 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:47.434 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.433+0000 7fbf59363640 10 monclient: shutdown 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.433+0000 7fbf59363640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.433+0000 7fbf59363640 1 -- 172.21.15.134:0/503028088 >> v1:172.21.15.134:6789/0 conn(0x7fbf5413f710 legacy=0x7fbf541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.433+0000 7fbf59363640 1 -- 172.21.15.134:0/503028088 shutdown_connections 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.433+0000 7fbf59363640 1 -- 172.21.15.134:0/503028088 >> 172.21.15.134:0/503028088 conn(0x7fbf540b7ea0 msgr2=0x7fbf540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.433+0000 7fbf59363640 1 -- 172.21.15.134:0/503028088 shutdown_connections 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.433+0000 7fbf59363640 1 -- 172.21.15.134:0/503028088 wait complete. 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 10 monclient: build_initial_monmap 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 10 monclient: monmap: 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:47.437 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 1 librados: starting msgr at 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 1 librados: starting objecter 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 1 Processor -- start 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 1 -- start start 2024-03-25T17:38:47.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 1 librados: setting wanted keys 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 1 librados: calling monclient init 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.434+0000 7fbf59363640 10 monclient: init 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 10 monclient(hunting): picked mon.noname-b con 0x7fbf540a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 10 monclient(hunting): picked mon.noname-c con 0x7fbf541430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf540b7c80 con 0x7fbf5413f710 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5414e4f0 con 0x7fbf540a98f0 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5413caf0 con 0x7fbf541430c0 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:47.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.435+0000 7fbf59363640 10 monclient(hunting): authenticate will time out at 5880.698464s 2024-03-25T17:38:47.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.441+0000 7fbf4a7fc640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fbf5413f710 0x7fbf5414d6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:57436/0 (socket says 172.21.15.134:57436) 2024-03-25T17:38:47.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.441+0000 7fbf537fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fbf541430c0 0x7fbf5414dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:41530/0 (socket says 172.21.15.134:41530) 2024-03-25T17:38:47.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.441+0000 7fbf4a7fc640 1 -- 172.21.15.134:0/1543935681 learned_addr learned my addr 172.21.15.134:0/1543935681 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:47.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.444+0000 7fbf4bfff640 10 client.?.objecter ms_handle_connect 0x7fbf5413f710 2024-03-25T17:38:47.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.444+0000 7fbf4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:47.445 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.445+0000 7fbf4bfff640 10 client.?.objecter ms_handle_connect 0x7fbf541430c0 2024-03-25T17:38:47.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.445+0000 7fbf4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:47.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.450+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111470334 0 0) 0x7fbf540b7c80 con 0x7fbf5413f710 2024-03-25T17:38:47.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.450+0000 7fbf4bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:47.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.450+0000 7fbf4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:47.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.450+0000 7fbf4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:47.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.450+0000 7fbf4bfff640 10 monclient(hunting): my global_id is 4629 2024-03-25T17:38:47.451 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.450+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf34004eb0 con 0x7fbf5413f710 2024-03-25T17:38:47.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 10 client.?.objecter ms_handle_connect 0x7fbf540a98f0 2024-03-25T17:38:47.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:47.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622159727 0 0) 0x7fbf5413caf0 con 0x7fbf541430c0 2024-03-25T17:38:47.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:47.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:47.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:47.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 10 monclient(hunting): my global_id is 34613 2024-03-25T17:38:47.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.460+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf540b7c80 con 0x7fbf541430c0 2024-03-25T17:38:47.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.465+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331184255 0 0) 0x7fbf5414e4f0 con 0x7fbf540a98f0 2024-03-25T17:38:47.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.465+0000 7fbf4bfff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:47.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.465+0000 7fbf4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:47.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.465+0000 7fbf4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:47.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.465+0000 7fbf4bfff640 10 monclient(hunting): my global_id is 4660 2024-03-25T17:38:47.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.465+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5413caf0 con 0x7fbf540a98f0 2024-03-25T17:38:47.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.467+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1690112434 0 0) 0x7fbf34004eb0 con 0x7fbf5413f710 2024-03-25T17:38:47.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.467+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5414e4f0 con 0x7fbf5413f710 2024-03-25T17:38:47.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.467+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbf3c001ba0 con 0x7fbf5413f710 2024-03-25T17:38:47.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.467+0000 7fbf4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:47.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.467+0000 7fbf4bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:47.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.467+0000 7fbf4bfff640 10 monclient(hunting): dump: 2024-03-25T17:38:47.468 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:47.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.467+0000 7fbf4bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.474+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 45853979 0 0) 0x7fbf5413caf0 con 0x7fbf540a98f0 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.474+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf34004eb0 con 0x7fbf540a98f0 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.474+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbf44001860 con 0x7fbf540a98f0 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.474+0000 7fbf4bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.474+0000 7fbf4bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.474+0000 7fbf4bfff640 10 monclient(hunting): dump: 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:47.475 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:47.476 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:47.476 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:47.476 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:47.476 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1943280926 0 0) 0x7fbf5414e4f0 con 0x7fbf5413f710 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 >> v1:172.21.15.134:6790/0 conn(0x7fbf541430c0 legacy=0x7fbf5414dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 >> v1:172.21.15.159:6789/0 conn(0x7fbf540a98f0 legacy=0x7fbf5413c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 10 monclient: _finish_hunting 0 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 1 monclient: found mon.a 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5413b970 con 0x7fbf5413f710 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 10 monclient: _finish_auth 0 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 10 monclient: _check_auth_tickets 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:47.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 5 monclient: authenticate success, global_id 4629 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 10 monclient: _renew_subs 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5413bc50 con 0x7fbf5413f710 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 10 monclient: _renew_subs 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5413a510 con 0x7fbf5413f710 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.478+0000 7fbf59363640 1 librados: init done 2024-03-25T17:38:47.480 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.479+0000 7fbf59363640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:47.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.479+0000 7fbf59363640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:47.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.479+0000 7fbf59363640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:47.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.479+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf540627b0 con 0x7fbf5413f710 2024-03-25T17:38:47.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.701+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf3c004a90 con 0x7fbf5413f710 2024-03-25T17:38:47.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.701+0000 7fbf4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:47.702 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.701+0000 7fbf51ffb640 4 set_mon_vals no callback set 2024-03-25T17:38:47.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.769+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fbf3c004ea0 con 0x7fbf5413f710 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.769+0000 7fbf4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.769+0000 7fbf4bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.769+0000 7fbf4bfff640 10 monclient: dump: 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:47.770 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:47.771 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:47.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.769+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fbf3c005080 con 0x7fbf5413f710 2024-03-25T17:38:47.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.770+0000 7fbf52ffd640 10 monclient: get_auth_request con 0x7fbf3406df70 auth_method 0 2024-03-25T17:38:47.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.770+0000 7fbf4bfff640 10 client.4629.objecter ms_handle_connect 0x7fbf3406df70 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fbf3c054a50 con 0x7fbf5413f710 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 10 client.4629.objecter ms_dispatch 0x7fbf54000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf3c050560 con 0x7fbf5413f710 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:47.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:47.776+0000 7fbf4bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:48.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.023+0000 7fbf59363640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:48.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.023+0000 7fbf59363640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:48.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.023+0000 7fbf59363640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:38:48.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.023+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf5413d730 con 0x7fbf5413f710 2024-03-25T17:38:48.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.054+0000 7fbf4bfff640 1 -- 172.21.15.134:0/1543935681 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fbf3c052490 con 0x7fbf5413f710 2024-03-25T17:38:48.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.054+0000 7fbf4bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:48.055 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.054+0000 7fbf4bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:48.055 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:48.055 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.056+0000 7fbf59363640 10 librados: watch_flush enter 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.056+0000 7fbf59363640 10 librados: watch_flush exit 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.056+0000 7fbf59363640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.056+0000 7fbf59363640 10 client.4629.objecter successfully canceled tick 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 >> v1:172.21.15.134:6800/610837766 conn(0x7fbf3406df70 legacy=0x7fbf34070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 10 monclient: shutdown 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 >> v1:172.21.15.134:6789/0 conn(0x7fbf5413f710 legacy=0x7fbf5414d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 shutdown_connections 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 >> 172.21.15.134:0/1543935681 conn(0x7fbf540b7ea0 msgr2=0x7fbf54074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 shutdown_connections 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 1 -- 172.21.15.134:0/1543935681 wait complete. 2024-03-25T17:38:48.058 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:48.057+0000 7fbf59363640 1 librados: shutdown 2024-03-25T17:38:48.070 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:38:50.717 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:55.733 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.770 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:55.798 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.771 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:55.787 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.773 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:55.802 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.803 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:09:55.819 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.809 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:09:55.838 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.831 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:55.847 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.838 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:55.854 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:50.866 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:09:55.882 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:38:54.877 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:38:54.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.973+0000 7f4497855640 10 monclient: get_monmap_and_config 2024-03-25T17:38:54.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.973+0000 7f4497855640 10 monclient: build_initial_monmap 2024-03-25T17:38:54.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.974+0000 7f4497855640 10 monclient: monmap: 2024-03-25T17:38:54.975 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:38:54.975983+0000 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:38:54.975983+0000 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:38:54.976 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.974+0000 7f4497855640 1 Processor -- start 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.974+0000 7f4497855640 1 -- start start 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.974+0000 7f4497855640 10 monclient: init 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.975+0000 7f4497855640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.975+0000 7f4497855640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.975+0000 7f4497855640 10 monclient(hunting): picked mon.noname-a con 0x7f44900b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.975+0000 7f4497855640 10 monclient(hunting): picked mon.noname-b con 0x7f44900b1730 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.975+0000 7f4497855640 10 monclient(hunting): picked mon.noname-c con 0x7f449014be90 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.975+0000 7f4497855640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44900af940 con 0x7f44900b4d60 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.975+0000 7f4497855640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44900b0a70 con 0x7f449014be90 2024-03-25T17:38:54.978 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.976+0000 7f4497855640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44900b0d50 con 0x7f44900b1730 2024-03-25T17:38:54.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.976+0000 7f4497855640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:54.979 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.976+0000 7f4497855640 10 monclient(hunting): authenticate will time out at 5888.239789s 2024-03-25T17:38:55.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.999+0000 7f44955f1640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f44900b4d60 0x7f449013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:40582/0 (socket says 172.21.15.134:40582) 2024-03-25T17:38:55.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:54.999+0000 7f44955f1640 1 -- 172.21.15.134:0/2898261102 learned_addr learned my addr 172.21.15.134:0/2898261102 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:55.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.009+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117831619 0 0) 0x7f44900af940 con 0x7f44900b4d60 2024-03-25T17:38:55.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.009+0000 7f4487fff640 10 monclient(hunting): _negotiate 2024-03-25T17:38:55.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.009+0000 7f4487fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:55.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.009+0000 7f4487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:55.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.009+0000 7f4487fff640 10 monclient(hunting): my global_id is 4632 2024-03-25T17:38:55.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.009+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f447c003520 con 0x7f44900b4d60 2024-03-25T17:38:55.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727581744 0 0) 0x7f447c003520 con 0x7f44900b4d60 2024-03-25T17:38:55.024 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 >> v1:172.21.15.134:6790/0 conn(0x7f44900b1730 legacy=0x7f44900b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 >> v1:172.21.15.159:6789/0 conn(0x7f449014be90 legacy=0x7f449014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 10 monclient: _finish_hunting 0 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 1 monclient: found mon.noname-a 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44900b1030 con 0x7f44900b4d60 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 10 monclient: _finish_auth 0 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 10 monclient: _check_auth_tickets 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f448c001690 con 0x7f44900b4d60 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:55.025 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.022+0000 7f4487fff640 10 monclient: dump: 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.023+0000 7f4497855640 5 monclient: authenticate success, global_id 4632 2024-03-25T17:38:55.026 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.023+0000 7f4497855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:38:55.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.034+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f448c001860 con 0x7f44900b4d60 2024-03-25T17:38:55.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4487fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:55.036 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4487fff640 1 -- 172.21.15.134:0/2898261102 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f448c001cb0 con 0x7f44900b4d60 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 10 monclient: get_monmap_and_config success 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 4 set_mon_vals no callback set 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4487fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4487fff640 10 monclient: dump: 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:55.037 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:55.038 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:55.038 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:55.038 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:55.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 10 monclient: shutdown 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 1 -- 172.21.15.134:0/2898261102 >> v1:172.21.15.134:6789/0 conn(0x7f44900b4d60 legacy=0x7f449013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 1 -- 172.21.15.134:0/2898261102 shutdown_connections 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 1 -- 172.21.15.134:0/2898261102 >> 172.21.15.134:0/2898261102 conn(0x7f44900617a0 msgr2=0x7f4490061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 1 -- 172.21.15.134:0/2898261102 shutdown_connections 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.035+0000 7f4497855640 1 -- 172.21.15.134:0/2898261102 wait complete. 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.036+0000 7f4497855640 10 monclient: build_initial_monmap 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.036+0000 7f4497855640 10 monclient: monmap: 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:38:55.040 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 librados: starting msgr at 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 librados: starting objecter 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 Processor -- start 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 -- start start 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 librados: setting wanted keys 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 librados: calling monclient init 2024-03-25T17:38:55.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient: init 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient: _reopen_session rank -1 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient(hunting): picked mon.noname-b con 0x7f44900b1730 addr v1:172.21.15.159:6789/0 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient(hunting): picked mon.noname-a con 0x7f44900b4d60 addr v1:172.21.15.134:6789/0 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient(hunting): picked mon.noname-c con 0x7f449014be90 addr v1:172.21.15.134:6790/0 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44901385c0 con 0x7f44900b4d60 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4490135f60 con 0x7f44900b1730 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44901360f0 con 0x7f449014be90 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient(hunting): _renew_subs 2024-03-25T17:38:55.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.037+0000 7f4497855640 10 monclient(hunting): authenticate will time out at 5888.301686s 2024-03-25T17:38:55.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.055+0000 7f4494df0640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f44900b4d60 0x7f449012fef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:40618/0 (socket says 172.21.15.134:40618) 2024-03-25T17:38:55.057 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.055+0000 7f4494df0640 1 -- 172.21.15.134:0/3205773913 learned_addr learned my addr 172.21.15.134:0/3205773913 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:38:55.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.058+0000 7f4485ffb640 10 client.?.objecter ms_handle_connect 0x7f44900b4d60 2024-03-25T17:38:55.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.058+0000 7f4485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:55.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.058+0000 7f4485ffb640 10 client.?.objecter ms_handle_connect 0x7f44900b1730 2024-03-25T17:38:55.060 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.058+0000 7f4485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:55.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.059+0000 7f4485ffb640 10 client.?.objecter ms_handle_connect 0x7f449014be90 2024-03-25T17:38:55.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.059+0000 7f4485ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:38:55.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.074+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385616698 0 0) 0x7f44901385c0 con 0x7f44900b4d60 2024-03-25T17:38:55.075 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.074+0000 7f4485ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:55.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.074+0000 7f4485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:55.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.074+0000 7f4485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:55.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.074+0000 7f4485ffb640 10 monclient(hunting): my global_id is 4635 2024-03-25T17:38:55.076 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.074+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4474004eb0 con 0x7f44900b4d60 2024-03-25T17:38:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.080+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591496638 0 0) 0x7f44901360f0 con 0x7f449014be90 2024-03-25T17:38:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.080+0000 7f4485ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.080+0000 7f4485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.080+0000 7f4485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.080+0000 7f4485ffb640 10 monclient(hunting): my global_id is 34616 2024-03-25T17:38:55.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.080+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44901385c0 con 0x7f449014be90 2024-03-25T17:38:55.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.081+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733469492 0 0) 0x7f4490135f60 con 0x7f44900b1730 2024-03-25T17:38:55.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.081+0000 7f4485ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:38:55.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.081+0000 7f4485ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:38:55.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.081+0000 7f4485ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:38:55.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.081+0000 7f4485ffb640 10 monclient(hunting): my global_id is 4663 2024-03-25T17:38:55.084 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.082+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44901360f0 con 0x7f44900b1730 2024-03-25T17:38:55.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.092+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2651613648 0 0) 0x7f44901360f0 con 0x7f44900b1730 2024-03-25T17:38:55.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.092+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4490135f60 con 0x7f44900b1730 2024-03-25T17:38:55.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.092+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f448c001b70 con 0x7f44900b1730 2024-03-25T17:38:55.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.092+0000 7f4485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.092+0000 7f4485ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.092+0000 7f4485ffb640 10 monclient(hunting): dump: 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:55.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.092+0000 7f4485ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:38:55.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.093+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 426183553 0 0) 0x7f44901385c0 con 0x7f449014be90 2024-03-25T17:38:55.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.093+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44901360f0 con 0x7f449014be90 2024-03-25T17:38:55.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.093+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4488001a70 con 0x7f449014be90 2024-03-25T17:38:55.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.093+0000 7f4485ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:55.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.093+0000 7f4485ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:38:55.096 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.093+0000 7f4485ffb640 10 monclient(hunting): dump: 2024-03-25T17:38:55.096 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:55.097 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:55.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1939668087 0 0) 0x7f4490135f60 con 0x7f44900b1730 2024-03-25T17:38:55.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 >> v1:172.21.15.134:6790/0 conn(0x7f449014be90 legacy=0x7f44901e2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:55.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 >> v1:172.21.15.134:6789/0 conn(0x7f44900b4d60 legacy=0x7f449012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 1 monclient: found mon.b 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490143900 con 0x7f44900b1730 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 10 monclient: _finish_auth 0 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4485ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4497855640 5 monclient: authenticate success, global_id 4663 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4497855640 10 monclient: _renew_subs 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4497855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.101+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44901d82b0 con 0x7f44900b1730 2024-03-25T17:38:55.104 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.102+0000 7f4497855640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:38:55.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.102+0000 7f4497855640 10 monclient: _renew_subs 2024-03-25T17:38:55.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.102+0000 7f4497855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:55.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.102+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44901307e0 con 0x7f44900b1730 2024-03-25T17:38:55.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.102+0000 7f4497855640 1 librados: init done 2024-03-25T17:38:55.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.103+0000 7f4497855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:55.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.103+0000 7f4497855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:55.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.103+0000 7f4497855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:55.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.103+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44900b22f0 con 0x7f44900b1730 2024-03-25T17:38:55.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.406+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f448c001cd0 con 0x7f44900b1730 2024-03-25T17:38:55.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.406+0000 7f4485ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:38:55.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.406+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f448c001250 con 0x7f44900b1730 2024-03-25T17:38:55.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.406+0000 7f4485ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:38:55.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.406+0000 7f4485ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.406+0000 7f4485ffb640 10 monclient: dump: 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:38:55.409 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.406+0000 7f44877fe640 4 set_mon_vals no callback set 2024-03-25T17:38:55.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.407+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f448c01aa60 con 0x7f44900b1730 2024-03-25T17:38:55.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.408+0000 7f4494df0640 10 monclient: get_auth_request con 0x7f447406df80 auth_method 0 2024-03-25T17:38:55.410 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.408+0000 7f4485ffb640 10 client.4663.objecter ms_handle_connect 0x7f447406df80 2024-03-25T17:38:55.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.412+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f448c054660 con 0x7f44900b1730 2024-03-25T17:38:55.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.412+0000 7f4485ffb640 10 client.4663.objecter ms_dispatch 0x7f4490000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:38:55.414 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.412+0000 7f4485ffb640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:38:55.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.412+0000 7f4485ffb640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:38:55.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.413+0000 7f4485ffb640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T17:38:55.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.413+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f448c04ffa0 con 0x7f44900b1730 2024-03-25T17:38:55.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.413+0000 7f4485ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:38:55.415 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.413+0000 7f4485ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:38:55.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.664+0000 7f4497855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:55.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.664+0000 7f4497855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:55.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.664+0000 7f4497855640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:38:55.666 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.664+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44901383c0 con 0x7f44900b1730 2024-03-25T17:38:55.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.694+0000 7f4485ffb640 1 -- 172.21.15.134:0/3205773913 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f448c0520a0 con 0x7f44900b1730 2024-03-25T17:38:55.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.694+0000 7f4485ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:38:55.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.694+0000 7f4485ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:38:55.696 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:38:55.696 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.697+0000 7f4497855640 10 librados: watch_flush enter 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.697+0000 7f4497855640 10 librados: watch_flush exit 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.697+0000 7f4497855640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.697+0000 7f4497855640 10 client.4663.objecter successfully canceled tick 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.697+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 >> v1:172.21.15.134:6800/610837766 conn(0x7f447406df80 legacy=0x7f4474070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 10 monclient: shutdown 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 >> v1:172.21.15.159:6789/0 conn(0x7f44900b1730 legacy=0x7f44901d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 shutdown_connections 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 >> 172.21.15.134:0/3205773913 conn(0x7f44900617a0 msgr2=0x7f44900b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 shutdown_connections 2024-03-25T17:38:55.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 1 -- 172.21.15.134:0/3205773913 wait complete. 2024-03-25T17:38:55.701 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:38:55.698+0000 7f4497855640 1 librados: shutdown 2024-03-25T17:38:55.711 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:02.519 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.630+0000 7fafaa3fe640 10 monclient: get_monmap_and_config 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.630+0000 7fafaa3fe640 10 monclient: build_initial_monmap 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.630+0000 7fafaa3fe640 10 monclient: monmap: 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:02.631458+0000 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:02.631458+0000 2024-03-25T17:39:02.631 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:02.632 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:02.632 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:02.632 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:02.632 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:02.632 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:02.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.631+0000 7fafaa3fe640 1 Processor -- start 2024-03-25T17:39:02.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.631+0000 7fafaa3fe640 1 -- start start 2024-03-25T17:39:02.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.631+0000 7fafaa3fe640 10 monclient: init 2024-03-25T17:39:02.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:02.633 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 10 monclient(hunting): picked mon.noname-a con 0x7fafa40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 10 monclient(hunting): picked mon.noname-c con 0x7fafa413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 10 monclient(hunting): picked mon.noname-b con 0x7fafa41430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa4142520 con 0x7fafa40a98f0 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa40b28a0 con 0x7fafa413f710 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa40b2a30 con 0x7fafa41430c0 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:02.634 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.632+0000 7fafaa3fe640 10 monclient(hunting): authenticate will time out at 5895.894955s 2024-03-25T17:39:02.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.638+0000 7fafa3fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fafa40a98f0 0x7fafa40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:55602/0 (socket says 172.21.15.134:55602) 2024-03-25T17:39:02.638 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.638+0000 7fafa3fff640 1 -- 172.21.15.134:0/2577939716 learned_addr learned my addr 172.21.15.134:0/2577939716 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:02.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.663+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329316553 0 0) 0x7fafa40b28a0 con 0x7fafa413f710 2024-03-25T17:39:02.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.663+0000 7fafa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.663 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.663+0000 7fafa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.663+0000 7fafa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.663+0000 7fafa2ffd640 10 monclient(hunting): my global_id is 4666 2024-03-25T17:39:02.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.663+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c003520 con 0x7fafa413f710 2024-03-25T17:39:02.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.664+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4237923202 0 0) 0x7fafa4142520 con 0x7fafa40a98f0 2024-03-25T17:39:02.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.664+0000 7fafa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.664+0000 7fafa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.664+0000 7fafa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.664+0000 7fafa2ffd640 10 monclient(hunting): my global_id is 4638 2024-03-25T17:39:02.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.664+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa40b28a0 con 0x7fafa40a98f0 2024-03-25T17:39:02.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.680+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242399041 0 0) 0x7fafa40b2a30 con 0x7fafa41430c0 2024-03-25T17:39:02.680 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.680+0000 7fafa2ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.680+0000 7fafa2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.680+0000 7fafa2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.680+0000 7fafa2ffd640 10 monclient(hunting): my global_id is 34619 2024-03-25T17:39:02.681 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.680+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa4142520 con 0x7fafa41430c0 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.691+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2671393137 0 0) 0x7fafa40b28a0 con 0x7fafa40a98f0 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 >> v1:172.21.15.134:6790/0 conn(0x7fafa41430c0 legacy=0x7fafa414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 >> v1:172.21.15.159:6789/0 conn(0x7fafa413f710 legacy=0x7fafa41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 1 monclient: found mon.noname-a 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa40b44c0 con 0x7fafa40a98f0 2024-03-25T17:39:02.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7faf94001670 con 0x7fafa40a98f0 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafaa3fe640 5 monclient: authenticate success, global_id 4638 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafaa3fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafa2ffd640 10 monclient: dump: 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.694 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.695 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.695 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:02.695 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:02.695 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:02.695 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:02.695 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:02.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.692+0000 7fafaa3fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf940017d0 con 0x7fafa40a98f0 2024-03-25T17:39:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafa2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafa2ffd640 1 -- 172.21.15.134:0/2577939716 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7faf94001be0 con 0x7fafa40a98f0 2024-03-25T17:39:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafaa3fe640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:02.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafaa3fe640 4 set_mon_vals no callback set 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafa2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.716+0000 7fafa2ffd640 10 monclient: dump: 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:02.718 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:02.719 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:02.719 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.717+0000 7fafaa3fe640 10 monclient: shutdown 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.717+0000 7fafaa3fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.717+0000 7fafaa3fe640 1 -- 172.21.15.134:0/2577939716 >> v1:172.21.15.134:6789/0 conn(0x7fafa40a98f0 legacy=0x7fafa40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.717+0000 7fafaa3fe640 1 -- 172.21.15.134:0/2577939716 shutdown_connections 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.717+0000 7fafaa3fe640 1 -- 172.21.15.134:0/2577939716 >> 172.21.15.134:0/2577939716 conn(0x7fafa40b7ea0 msgr2=0x7fafa40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.717+0000 7fafaa3fe640 1 -- 172.21.15.134:0/2577939716 shutdown_connections 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.717+0000 7fafaa3fe640 1 -- 172.21.15.134:0/2577939716 wait complete. 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.718+0000 7fafaa3fe640 10 monclient: build_initial_monmap 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.718+0000 7fafaa3fe640 10 monclient: monmap: 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:02.721 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.718+0000 7fafaa3fe640 1 librados: starting msgr at 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.718+0000 7fafaa3fe640 1 librados: starting objecter 2024-03-25T17:39:02.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.718+0000 7fafaa3fe640 1 Processor -- start 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 1 -- start start 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 1 librados: setting wanted keys 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 1 librados: calling monclient init 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient: init 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient(hunting): picked mon.noname-a con 0x7fafa40a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient(hunting): picked mon.noname-c con 0x7fafa413f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient(hunting): picked mon.noname-b con 0x7fafa41430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa4139720 con 0x7fafa40a98f0 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa4138af0 con 0x7fafa41430c0 2024-03-25T17:39:02.723 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa4138d60 con 0x7fafa413f710 2024-03-25T17:39:02.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:02.724 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.719+0000 7fafaa3fe640 10 monclient(hunting): authenticate will time out at 5895.982524s 2024-03-25T17:39:02.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.726+0000 7fafa37fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fafa413f710 0x7fafa41372d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:36772/0 (socket says 172.21.15.134:36772) 2024-03-25T17:39:02.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.726+0000 7fafa37fe640 1 -- 172.21.15.134:0/850175711 learned_addr learned my addr 172.21.15.134:0/850175711 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:02.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.727+0000 7fafa0ff9640 10 client.?.objecter ms_handle_connect 0x7fafa413f710 2024-03-25T17:39:02.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.727+0000 7fafa0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:02.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.737+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661519038 0 0) 0x7fafa4138d60 con 0x7fafa413f710 2024-03-25T17:39:02.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.737+0000 7fafa0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.737+0000 7fafa0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.737+0000 7fafa0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.737+0000 7fafa0ff9640 10 monclient(hunting): my global_id is 34622 2024-03-25T17:39:02.738 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.737+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf90004eb0 con 0x7fafa413f710 2024-03-25T17:39:02.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.739+0000 7fafa0ff9640 10 client.?.objecter ms_handle_connect 0x7fafa41430c0 2024-03-25T17:39:02.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.739+0000 7fafa0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:02.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.748+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 574391373 0 0) 0x7faf90004eb0 con 0x7fafa413f710 2024-03-25T17:39:02.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.748+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafa4138d60 con 0x7fafa413f710 2024-03-25T17:39:02.749 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.749+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7faf98001990 con 0x7fafa413f710 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.749+0000 7fafa0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.749+0000 7fafa0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.749+0000 7fafa0ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:02.750 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:02.751 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:02.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.749+0000 7fafa0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:02.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.749+0000 7fafa0ff9640 10 client.?.objecter ms_handle_connect 0x7fafa40a98f0 2024-03-25T17:39:02.751 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.749+0000 7fafa0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:02.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.757+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206179218 0 0) 0x7fafa4138af0 con 0x7fafa41430c0 2024-03-25T17:39:02.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.757+0000 7fafa0ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:02.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.757+0000 7fafa0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:02.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.757+0000 7fafa0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:02.757 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.757+0000 7fafa0ff9640 10 monclient(hunting): my global_id is 4669 2024-03-25T17:39:02.758 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.757+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf90004eb0 con 0x7fafa41430c0 2024-03-25T17:39:02.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 253443732 0 0) 0x7fafa4138d60 con 0x7fafa413f710 2024-03-25T17:39:02.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 >> v1:172.21.15.159:6789/0 conn(0x7fafa41430c0 legacy=0x7fafa40b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 >> v1:172.21.15.134:6789/0 conn(0x7fafa40a98f0 legacy=0x7fafa4136690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:02.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:39:02.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 1 monclient: found mon.c 2024-03-25T17:39:02.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:02.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa41379e0 con 0x7fafa413f710 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 10 monclient: _finish_auth 0 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafa0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 5 monclient: authenticate success, global_id 34622 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 10 monclient: _renew_subs 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa41da8f0 con 0x7fafa413f710 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 10 client.34622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 10 monclient: _renew_subs 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa413c380 con 0x7fafa413f710 2024-03-25T17:39:02.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.762+0000 7fafaa3fe640 1 librados: init done 2024-03-25T17:39:02.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.763+0000 7fafaa3fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.763+0000 7fafaa3fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:02.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.763+0000 7fafaa3fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:39:02.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.763+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa4136df0 con 0x7fafa413f710 2024-03-25T17:39:02.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.981+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf98001c90 con 0x7fafa413f710 2024-03-25T17:39:02.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.981+0000 7fafa0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:02.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.981+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7faf98003a60 con 0x7fafa413f710 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.981+0000 7fafa0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.981+0000 7fafa0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.981+0000 7fafa0ff9640 10 monclient: dump: 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:02.982 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:02.981+0000 7fafa27fc640 4 set_mon_vals no callback set 2024-03-25T17:39:03.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.061+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7faf98003c40 con 0x7fafa413f710 2024-03-25T17:39:03.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.062+0000 7fafa3fff640 10 monclient: get_auth_request con 0x7faf9006dce0 auth_method 0 2024-03-25T17:39:03.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.062+0000 7fafa0ff9640 10 client.34622.objecter ms_handle_connect 0x7faf9006dce0 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7faf980548e0 con 0x7fafa413f710 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 10 client.34622.objecter ms_dispatch 0x7fafa4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 3 client.34622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 3 client.34622.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 20 client.34622.objecter dump_active .. 0 homeless 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf98050220 con 0x7fafa413f710 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:03.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.070+0000 7fafa0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:03.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.321+0000 7fafaa3fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:03.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.321+0000 7fafaa3fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:03.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.321+0000 7fafaa3fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:39:03.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.321+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafa40627b0 con 0x7fafa413f710 2024-03-25T17:39:03.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.343+0000 7fafa0ff9640 1 -- 172.21.15.134:0/850175711 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7faf98052320 con 0x7fafa413f710 2024-03-25T17:39:03.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.343+0000 7fafa0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:03.344 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.343+0000 7fafa0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:03.344 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:03.345 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 10 librados: watch_flush enter 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 10 librados: watch_flush exit 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 20 client.34622.objecter shutdown clearing up homeless session... 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 10 client.34622.objecter successfully canceled tick 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 >> v1:172.21.15.134:6800/610837766 conn(0x7faf9006dce0 legacy=0x7faf900701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 10 monclient: shutdown 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:03.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 >> v1:172.21.15.134:6790/0 conn(0x7fafa413f710 legacy=0x7fafa41372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:03.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 shutdown_connections 2024-03-25T17:39:03.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 >> 172.21.15.134:0/850175711 conn(0x7fafa40b7ea0 msgr2=0x7fafa4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:03.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 shutdown_connections 2024-03-25T17:39:03.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 1 -- 172.21.15.134:0/850175711 wait complete. 2024-03-25T17:39:03.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:03.346+0000 7fafaa3fe640 1 librados: shutdown 2024-03-25T17:39:03.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:10.168 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.282+0000 7fc9d71d6640 10 monclient: get_monmap_and_config 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.282+0000 7fc9d71d6640 10 monclient: build_initial_monmap 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.282+0000 7fc9d71d6640 10 monclient: monmap: 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:10.283688+0000 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:10.283688+0000 2024-03-25T17:39:10.284 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.283+0000 7fc9d71d6640 1 Processor -- start 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.283+0000 7fc9d71d6640 1 -- start start 2024-03-25T17:39:10.285 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.283+0000 7fc9d71d6640 10 monclient: init 2024-03-25T17:39:10.286 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9d00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9d013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9d01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d0142520 con 0x7fc9d01430c0 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d00b28a0 con 0x7fc9d00a98f0 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d00b2a30 con 0x7fc9d013f710 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:10.287 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.284+0000 7fc9d71d6640 10 monclient(hunting): authenticate will time out at 5903.547374s 2024-03-25T17:39:10.288 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.288+0000 7fc9cffff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fc9d013f710 0x7fc9d01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:41242/0 (socket says 172.21.15.134:41242) 2024-03-25T17:39:10.289 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.288+0000 7fc9cffff640 1 -- 172.21.15.134:0/1117105742 learned_addr learned my addr 172.21.15.134:0/1117105742 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:10.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.296+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206540177 0 0) 0x7fc9d00b2a30 con 0x7fc9d013f710 2024-03-25T17:39:10.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.296+0000 7fc9cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:10.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.296+0000 7fc9cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:10.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.296+0000 7fc9cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:10.297 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.296+0000 7fc9cf7fe640 10 monclient(hunting): my global_id is 34625 2024-03-25T17:39:10.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.296+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c0003520 con 0x7fc9d013f710 2024-03-25T17:39:10.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.297+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3642739148 0 0) 0x7fc9d0142520 con 0x7fc9d01430c0 2024-03-25T17:39:10.298 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.297+0000 7fc9cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:10.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.297+0000 7fc9cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:10.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.297+0000 7fc9cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:10.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.297+0000 7fc9cf7fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T17:39:10.299 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.297+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d00b2a30 con 0x7fc9d01430c0 2024-03-25T17:39:10.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.308+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781868352 0 0) 0x7fc9d00b2a30 con 0x7fc9d01430c0 2024-03-25T17:39:10.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 >> v1:172.21.15.134:6790/0 conn(0x7fc9d013f710 legacy=0x7fc9d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:10.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 >> v1:172.21.15.159:6789/0 conn(0x7fc9d00a98f0 legacy=0x7fc9d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:10.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:10.310 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 1 monclient: found mon.noname-a 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d00b44c0 con 0x7fc9d01430c0 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc9c80019f0 con 0x7fc9d01430c0 2024-03-25T17:39:10.311 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9cf7fe640 10 monclient: dump: 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:10.312 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:10.313 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:10.313 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:10.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9d71d6640 5 monclient: authenticate success, global_id 4644 2024-03-25T17:39:10.313 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.309+0000 7fc9d71d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:10.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9c8001b50 con 0x7fc9d01430c0 2024-03-25T17:39:10.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:10.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9cf7fe640 1 -- 172.21.15.134:0/1117105742 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc9c8001f20 con 0x7fc9d01430c0 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9cf7fe640 10 monclient: dump: 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:10.317 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9d71d6640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.314+0000 7fc9d71d6640 4 set_mon_vals no callback set 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.315+0000 7fc9d71d6640 10 monclient: shutdown 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.315+0000 7fc9d71d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:10.318 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.315+0000 7fc9d71d6640 1 -- 172.21.15.134:0/1117105742 >> v1:172.21.15.134:6789/0 conn(0x7fc9d01430c0 legacy=0x7fc9d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:10.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.315+0000 7fc9d71d6640 1 -- 172.21.15.134:0/1117105742 shutdown_connections 2024-03-25T17:39:10.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.315+0000 7fc9d71d6640 1 -- 172.21.15.134:0/1117105742 >> 172.21.15.134:0/1117105742 conn(0x7fc9d00b7ea0 msgr2=0x7fc9d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:10.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.315+0000 7fc9d71d6640 1 -- 172.21.15.134:0/1117105742 shutdown_connections 2024-03-25T17:39:10.319 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.315+0000 7fc9d71d6640 1 -- 172.21.15.134:0/1117105742 wait complete. 2024-03-25T17:39:10.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 10 monclient: build_initial_monmap 2024-03-25T17:39:10.321 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 10 monclient: monmap: 2024-03-25T17:39:10.321 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:10.321 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:10.321 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:10.321 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 1 librados: starting msgr at 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 1 librados: starting objecter 2024-03-25T17:39:10.322 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 1 Processor -- start 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 1 -- start start 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 1 librados: setting wanted keys 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 1 librados: calling monclient init 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.316+0000 7fc9d71d6640 10 monclient: init 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9d00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9d013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:10.323 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9d01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:10.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d0139890 con 0x7fc9d01430c0 2024-03-25T17:39:10.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d01dba10 con 0x7fc9d013f710 2024-03-25T17:39:10.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d013ace0 con 0x7fc9d00a98f0 2024-03-25T17:39:10.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:10.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.317+0000 7fc9d71d6640 10 monclient(hunting): authenticate will time out at 5903.580595s 2024-03-25T17:39:10.324 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.324+0000 7fc9cffff640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc9d013f710 0x7fc9d01dabf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:57606/0 (socket says 172.21.15.134:57606) 2024-03-25T17:39:10.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.324+0000 7fc9cffff640 1 -- 172.21.15.134:0/3861547380 learned_addr learned my addr 172.21.15.134:0/3861547380 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:10.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.324+0000 7fc9cd7fa640 10 client.?.objecter ms_handle_connect 0x7fc9d013f710 2024-03-25T17:39:10.325 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.324+0000 7fc9cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:10.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.334+0000 7fc9cd7fa640 10 client.?.objecter ms_handle_connect 0x7fc9d00a98f0 2024-03-25T17:39:10.335 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.334+0000 7fc9cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:10.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.337+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435402787 0 0) 0x7fc9d01dba10 con 0x7fc9d013f710 2024-03-25T17:39:10.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.337+0000 7fc9cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:10.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.337+0000 7fc9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:10.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.337+0000 7fc9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:10.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.337+0000 7fc9cd7fa640 10 monclient(hunting): my global_id is 4675 2024-03-25T17:39:10.338 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.337+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b4004eb0 con 0x7fc9d013f710 2024-03-25T17:39:10.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.340+0000 7fc9cd7fa640 10 client.?.objecter ms_handle_connect 0x7fc9d01430c0 2024-03-25T17:39:10.341 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.340+0000 7fc9cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:10.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077598675 0 0) 0x7fc9b4004eb0 con 0x7fc9d013f710 2024-03-25T17:39:10.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9d01dba10 con 0x7fc9d013f710 2024-03-25T17:39:10.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc9bc001990 con 0x7fc9d013f710 2024-03-25T17:39:10.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:10.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): dump: 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:10.347 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689692116 0 0) 0x7fc9d013ace0 con 0x7fc9d00a98f0 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 10 monclient(hunting): my global_id is 34628 2024-03-25T17:39:10.348 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.345+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b4004eb0 con 0x7fc9d00a98f0 2024-03-25T17:39:10.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.356+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174130814 0 0) 0x7fc9d0139890 con 0x7fc9d01430c0 2024-03-25T17:39:10.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.356+0000 7fc9cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T17:39:10.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.356+0000 7fc9cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:10.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.356+0000 7fc9cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:10.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.356+0000 7fc9cd7fa640 10 monclient(hunting): my global_id is 4647 2024-03-25T17:39:10.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.356+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d013ace0 con 0x7fc9d01430c0 2024-03-25T17:39:10.360 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.358+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2981804616 0 0) 0x7fc9d01dba10 con 0x7fc9d013f710 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 >> v1:172.21.15.134:6790/0 conn(0x7fc9d00a98f0 legacy=0x7fc9d0139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 >> v1:172.21.15.134:6789/0 conn(0x7fc9d01430c0 legacy=0x7fc9d01db300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 1 monclient: found mon.b 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d013afc0 con 0x7fc9d013f710 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 10 monclient: _finish_auth 0 2024-03-25T17:39:10.361 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 5 monclient: authenticate success, global_id 4675 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 10 monclient: _renew_subs 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d013b2a0 con 0x7fc9d013f710 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 10 monclient: _renew_subs 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:10.362 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d00b7b50 con 0x7fc9d013f710 2024-03-25T17:39:10.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.359+0000 7fc9d71d6640 1 librados: init done 2024-03-25T17:39:10.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.360+0000 7fc9d71d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:10.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.360+0000 7fc9d71d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:10.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.360+0000 7fc9d71d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:10.363 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.360+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d00627b0 con 0x7fc9d013f710 2024-03-25T17:39:10.645 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9bc0047e0 con 0x7fc9d013f710 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc9bc004bb0 con 0x7fc9d013f710 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9cd7fa640 10 monclient: dump: 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:10.646 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:10.647 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:10.647 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:10.647 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:10.647 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:10.647 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:10.647 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9ceffd640 4 set_mon_vals no callback set 2024-03-25T17:39:10.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.645+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fc9bc004dd0 con 0x7fc9d013f710 2024-03-25T17:39:10.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.646+0000 7fc9d4f72640 10 monclient: get_auth_request con 0x7fc9b406de80 auth_method 0 2024-03-25T17:39:10.648 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.646+0000 7fc9cd7fa640 10 client.4675.objecter ms_handle_connect 0x7fc9b406de80 2024-03-25T17:39:10.651 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.651+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fc9bc0548d0 con 0x7fc9d013f710 2024-03-25T17:39:10.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.651+0000 7fc9cd7fa640 10 client.4675.objecter ms_dispatch 0x7fc9d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:10.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.651+0000 7fc9cd7fa640 3 client.4675.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:10.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.651+0000 7fc9cd7fa640 3 client.4675.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:10.652 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.651+0000 7fc9cd7fa640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T17:39:10.653 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.653+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9bc0503e0 con 0x7fc9d013f710 2024-03-25T17:39:10.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.653+0000 7fc9cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:10.654 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.653+0000 7fc9cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:10.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.895+0000 7fc9d71d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:10.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.895+0000 7fc9d71d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:10.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.895+0000 7fc9d71d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:10.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.895+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9d00b7120 con 0x7fc9d013f710 2024-03-25T17:39:10.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.946+0000 7fc9cd7fa640 1 -- 172.21.15.134:0/3861547380 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7fc9d00b7120 con 0x7fc9d013f710 2024-03-25T17:39:10.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.946+0000 7fc9cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:10.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.946+0000 7fc9cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:10.947 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:10.947 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.948+0000 7fc9d71d6640 10 librados: watch_flush enter 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 10 librados: watch_flush exit 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 10 client.4675.objecter successfully canceled tick 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 >> v1:172.21.15.134:6800/610837766 conn(0x7fc9b406de80 legacy=0x7fc9b4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 10 monclient: shutdown 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 >> v1:172.21.15.159:6789/0 conn(0x7fc9d013f710 legacy=0x7fc9d01dabf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:10.950 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 shutdown_connections 2024-03-25T17:39:10.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 >> 172.21.15.134:0/3861547380 conn(0x7fc9d00b7ea0 msgr2=0x7fc9d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:10.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 shutdown_connections 2024-03-25T17:39:10.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 1 -- 172.21.15.134:0/3861547380 wait complete. 2024-03-25T17:39:10.951 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:10.949+0000 7fc9d71d6640 1 librados: shutdown 2024-03-25T17:39:10.962 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:15.834 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:20.850 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.838 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:20.854 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.873 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:20.890 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.884 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:20.913 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.899 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:20.928 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.912 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:20.929 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.917 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:20.934 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.920 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:20.949 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:15.939 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:20.955 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:17.770 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:17.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.884+0000 7f5ace0c9640 10 monclient: get_monmap_and_config 2024-03-25T17:39:17.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.884+0000 7f5ace0c9640 10 monclient: build_initial_monmap 2024-03-25T17:39:17.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.884+0000 7f5ace0c9640 10 monclient: monmap: 2024-03-25T17:39:17.885 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:17.885 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:17.885 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:17.885335+0000 2024-03-25T17:39:17.886 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:17.885335+0000 2024-03-25T17:39:17.886 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:17.886 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:17.886 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:17.886 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:17.886 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:17.886 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:17.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.885+0000 7f5ace0c9640 1 Processor -- start 2024-03-25T17:39:17.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.885+0000 7f5ace0c9640 1 -- start start 2024-03-25T17:39:17.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.885+0000 7f5ace0c9640 10 monclient: init 2024-03-25T17:39:17.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.885+0000 7f5ace0c9640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:17.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.885+0000 7f5ace0c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.885+0000 7f5ace0c9640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac80a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.885+0000 7f5ace0c9640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.886+0000 7f5ace0c9640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac81430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.886+0000 7f5ace0c9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac8142520 con 0x7f5ac81430c0 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.886+0000 7f5ace0c9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80b28a0 con 0x7f5ac813f710 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.886+0000 7f5ace0c9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac80b2a30 con 0x7f5ac80a98f0 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.886+0000 7f5ace0c9640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.886+0000 7f5ace0c9640 10 monclient(hunting): authenticate will time out at 5911.148913s 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.887+0000 7f5ac7fff640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f5ac81430c0 0x7f5ac814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34008/0 (socket says 172.21.15.134:34008) 2024-03-25T17:39:17.889 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.887+0000 7f5ac7fff640 1 -- 172.21.15.134:0/2210490953 learned_addr learned my addr 172.21.15.134:0/2210490953 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:17.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.894+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935681154 0 0) 0x7f5ac8142520 con 0x7f5ac81430c0 2024-03-25T17:39:17.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.894+0000 7f5ac67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:39:17.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.894+0000 7f5ac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:17.895 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.894+0000 7f5ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:17.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.894+0000 7f5ac67fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T17:39:17.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.894+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab0003520 con 0x7f5ac81430c0 2024-03-25T17:39:17.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 788183453 0 0) 0x7f5ab0003520 con 0x7f5ac81430c0 2024-03-25T17:39:17.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 >> v1:172.21.15.134:6790/0 conn(0x7f5ac80a98f0 legacy=0x7f5ac80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:17.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 >> v1:172.21.15.159:6789/0 conn(0x7f5ac813f710 legacy=0x7f5ac81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:17.914 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 10 monclient: _finish_hunting 0 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 1 monclient: found mon.noname-a 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac80b44c0 con 0x7f5ac81430c0 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 10 monclient: _finish_auth 0 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 10 monclient: _check_auth_tickets 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5ac0001a10 con 0x7f5ac81430c0 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:17.915 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ac67fc640 10 monclient: dump: 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:17.916 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:17.917 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:17.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ace0c9640 5 monclient: authenticate success, global_id 4650 2024-03-25T17:39:17.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.913+0000 7f5ace0c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:17.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ac0001b70 con 0x7f5ac81430c0 2024-03-25T17:39:17.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:17.923 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ac67fc640 1 -- 172.21.15.134:0/2210490953 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5ac0001fc0 con 0x7f5ac81430c0 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ace0c9640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ace0c9640 4 set_mon_vals no callback set 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ac67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.921+0000 7f5ac67fc640 10 monclient: dump: 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.924 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.922+0000 7f5ace0c9640 10 monclient: shutdown 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.922+0000 7f5ace0c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:17.925 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.922+0000 7f5ace0c9640 1 -- 172.21.15.134:0/2210490953 >> v1:172.21.15.134:6789/0 conn(0x7f5ac81430c0 legacy=0x7f5ac814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:17.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.922+0000 7f5ace0c9640 1 -- 172.21.15.134:0/2210490953 shutdown_connections 2024-03-25T17:39:17.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.922+0000 7f5ace0c9640 1 -- 172.21.15.134:0/2210490953 >> 172.21.15.134:0/2210490953 conn(0x7f5ac80b7ea0 msgr2=0x7f5ac80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:17.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.922+0000 7f5ace0c9640 1 -- 172.21.15.134:0/2210490953 shutdown_connections 2024-03-25T17:39:17.926 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.922+0000 7f5ace0c9640 1 -- 172.21.15.134:0/2210490953 wait complete. 2024-03-25T17:39:17.927 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.923+0000 7f5ace0c9640 10 monclient: build_initial_monmap 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.923+0000 7f5ace0c9640 10 monclient: monmap: 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:17.928 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.924+0000 7f5ace0c9640 1 librados: starting msgr at 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.924+0000 7f5ace0c9640 1 librados: starting objecter 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.924+0000 7f5ace0c9640 1 Processor -- start 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.924+0000 7f5ace0c9640 1 -- start start 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.924+0000 7f5ace0c9640 1 librados: setting wanted keys 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.924+0000 7f5ace0c9640 1 librados: calling monclient init 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.924+0000 7f5ace0c9640 10 monclient: init 2024-03-25T17:39:17.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 10 monclient(hunting): picked mon.noname-b con 0x7f5ac80a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 10 monclient(hunting): picked mon.noname-a con 0x7f5ac813f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 10 monclient(hunting): picked mon.noname-c con 0x7f5ac81430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac813b9e0 con 0x7f5ac813f710 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac814dd30 con 0x7f5ac80a98f0 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac8135680 con 0x7f5ac81430c0 2024-03-25T17:39:17.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:17.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.925+0000 7f5ace0c9640 10 monclient(hunting): authenticate will time out at 5911.188166s 2024-03-25T17:39:17.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.937+0000 7f5ac6ffd640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f5ac813f710 0x7f5ac8134860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34030/0 (socket says 172.21.15.134:34030) 2024-03-25T17:39:17.938 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.937+0000 7f5ac6ffd640 1 -- 172.21.15.134:0/779310292 learned_addr learned my addr 172.21.15.134:0/779310292 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:17.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.938+0000 7f5aaffff640 10 client.?.objecter ms_handle_connect 0x7f5ac813f710 2024-03-25T17:39:17.939 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.938+0000 7f5aaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:17.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.943+0000 7f5aaffff640 10 client.?.objecter ms_handle_connect 0x7f5ac80a98f0 2024-03-25T17:39:17.944 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.943+0000 7f5aaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:17.946 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.946+0000 7f5aaffff640 10 client.?.objecter ms_handle_connect 0x7f5ac81430c0 2024-03-25T17:39:17.947 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.946+0000 7f5aaffff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:17.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.947+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825503581 0 0) 0x7f5ac813b9e0 con 0x7f5ac813f710 2024-03-25T17:39:17.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.947+0000 7f5aaffff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:17.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.947+0000 7f5aaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:17.948 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.947+0000 7f5aaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:17.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.947+0000 7f5aaffff640 10 monclient(hunting): my global_id is 4653 2024-03-25T17:39:17.949 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.947+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4004eb0 con 0x7f5ac813f710 2024-03-25T17:39:17.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.951+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763310121 0 0) 0x7f5ac814dd30 con 0x7f5ac80a98f0 2024-03-25T17:39:17.952 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.951+0000 7f5aaffff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.951+0000 7f5aaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.951+0000 7f5aaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.951+0000 7f5aaffff640 10 monclient(hunting): my global_id is 4681 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.951+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac813b9e0 con 0x7f5ac80a98f0 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.951+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589787099 0 0) 0x7f5ac8135680 con 0x7f5ac81430c0 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.952+0000 7f5aaffff640 10 monclient(hunting): _negotiate 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.952+0000 7f5aaffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.952+0000 7f5aaffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.952+0000 7f5aaffff640 10 monclient(hunting): my global_id is 34634 2024-03-25T17:39:17.953 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.952+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac814dd30 con 0x7f5ac81430c0 2024-03-25T17:39:17.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.956+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1312300621 0 0) 0x7f5ab4004eb0 con 0x7f5ac813f710 2024-03-25T17:39:17.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.956+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ac8135680 con 0x7f5ac813f710 2024-03-25T17:39:17.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.956+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5abc001a30 con 0x7f5ac813f710 2024-03-25T17:39:17.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.956+0000 7f5aaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:17.957 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.956+0000 7f5aaffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.956+0000 7f5aaffff640 10 monclient(hunting): dump: 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:17.958 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:17.959 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:17.959 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:17.959 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.956+0000 7f5aaffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:17.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.965+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4057124105 0 0) 0x7f5ac813b9e0 con 0x7f5ac80a98f0 2024-03-25T17:39:17.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.965+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab4004eb0 con 0x7f5ac80a98f0 2024-03-25T17:39:17.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.965+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5ab80018b0 con 0x7f5ac80a98f0 2024-03-25T17:39:17.966 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.965+0000 7f5aaffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.965+0000 7f5aaffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.965+0000 7f5aaffff640 10 monclient(hunting): dump: 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:17.967 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:17.968 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:17.968 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:17.974 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3246178103 0 0) 0x7f5ac8135680 con 0x7f5ac813f710 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 >> v1:172.21.15.134:6790/0 conn(0x7f5ac81430c0 legacy=0x7f5ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 >> v1:172.21.15.159:6789/0 conn(0x7f5ac80a98f0 legacy=0x7f5ac814e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 10 monclient: _finish_hunting 0 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 1 monclient: found mon.a 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac813be70 con 0x7f5ac813f710 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 10 monclient: _finish_auth 0 2024-03-25T17:39:17.975 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 10 monclient: _check_auth_tickets 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5aaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 5 monclient: authenticate success, global_id 4653 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 10 monclient: _renew_subs 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac814caa0 con 0x7f5ac813f710 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 10 monclient: _renew_subs 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac814c730 con 0x7f5ac813f710 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.973+0000 7f5ace0c9640 1 librados: init done 2024-03-25T17:39:17.976 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.974+0000 7f5ace0c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:17.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.974+0000 7f5ace0c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:17.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.974+0000 7f5ace0c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:17.977 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:17.974+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac813bb90 con 0x7f5ac813f710 2024-03-25T17:39:18.262 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.261+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5abc001c70 con 0x7f5ac813f710 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.261+0000 7f5aaffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.261+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f5abc0038c0 con 0x7f5ac813f710 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.261+0000 7f5aaffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.261+0000 7f5aaffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.261+0000 7f5aaffff640 10 monclient: dump: 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:18.263 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:18.264 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:18.264 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:18.264 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:18.264 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:18.264 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:18.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.261+0000 7f5ac5ffb640 4 set_mon_vals no callback set 2024-03-25T17:39:18.264 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.262+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f5abc003ae0 con 0x7f5ac813f710 2024-03-25T17:39:18.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.262+0000 7f5ac77fe640 10 monclient: get_auth_request con 0x7f5ab406df70 auth_method 0 2024-03-25T17:39:18.265 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.263+0000 7f5aaffff640 10 client.4653.objecter ms_handle_connect 0x7f5ab406df70 2024-03-25T17:39:18.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7f5abc054700 con 0x7f5ac813f710 2024-03-25T17:39:18.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 10 client.4653.objecter ms_dispatch 0x7f5ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:18.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:18.269 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:18.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T17:39:18.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5abc050210 con 0x7f5ac813f710 2024-03-25T17:39:18.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:18.270 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.268+0000 7f5aaffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:18.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.497+0000 7f5ace0c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:18.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.498+0000 7f5ace0c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:18.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.498+0000 7f5ace0c9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:18.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.498+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ac814ce40 con 0x7f5ac813f710 2024-03-25T17:39:18.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.532+0000 7f5aaffff640 1 -- 172.21.15.134:0/779310292 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1595631655) 0x7f5abc052140 con 0x7f5ac813f710 2024-03-25T17:39:18.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.532+0000 7f5aaffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:18.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.532+0000 7f5aaffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:18.533 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:18.533 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":10,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:29:28.292736+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:29:28.292736+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:18.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 10 librados: watch_flush enter 2024-03-25T17:39:18.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 10 librados: watch_flush exit 2024-03-25T17:39:18.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T17:39:18.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 10 client.4653.objecter successfully canceled tick 2024-03-25T17:39:18.536 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 >> v1:172.21.15.134:6800/610837766 conn(0x7f5ab406df70 legacy=0x7f5ab4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 10 monclient: shutdown 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 >> v1:172.21.15.134:6789/0 conn(0x7f5ac813f710 legacy=0x7f5ac8134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 shutdown_connections 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.534+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 >> 172.21.15.134:0/779310292 conn(0x7f5ac80b7ea0 msgr2=0x7f5ac8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.535+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 shutdown_connections 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.535+0000 7f5ace0c9640 1 -- 172.21.15.134:0/779310292 wait complete. 2024-03-25T17:39:18.537 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:18.535+0000 7f5ace0c9640 1 librados: shutdown 2024-03-25T17:39:18.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:25.354 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.474+0000 7fc2ac8c1640 10 monclient: get_monmap_and_config 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.474+0000 7fc2ac8c1640 10 monclient: build_initial_monmap 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.474+0000 7fc2ac8c1640 10 monclient: monmap: 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:25.475393+0000 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:25.475393+0000 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:25.475 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:25.476 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 1 Processor -- start 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 1 -- start start 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 10 monclient: init 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a80a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc2a81430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.475+0000 7fc2ac8c1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a8142520 con 0x7fc2a80a98f0 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.476+0000 7fc2ac8c1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80b28a0 con 0x7fc2a81430c0 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.476+0000 7fc2ac8c1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80b2a30 con 0x7fc2a813f710 2024-03-25T17:39:25.477 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.476+0000 7fc2ac8c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:25.478 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.476+0000 7fc2ac8c1640 10 monclient(hunting): authenticate will time out at 5918.738847s 2024-03-25T17:39:25.490 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.490+0000 7fc2a6d9d640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fc2a81430c0 0x7fc2a814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:44700/0 (socket says 172.21.15.134:44700) 2024-03-25T17:39:25.491 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.490+0000 7fc2a6d9d640 1 -- 172.21.15.134:0/3242973804 learned_addr learned my addr 172.21.15.134:0/3242973804 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:25.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.509+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90024598 0 0) 0x7fc2a80b28a0 con 0x7fc2a81430c0 2024-03-25T17:39:25.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.509+0000 7fc2a559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:25.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.509+0000 7fc2a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:25.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.509+0000 7fc2a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:25.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.509+0000 7fc2a559a640 10 monclient(hunting): my global_id is 4684 2024-03-25T17:39:25.510 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.509+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc290003520 con 0x7fc2a81430c0 2024-03-25T17:39:25.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.515+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289650274 0 0) 0x7fc2a8142520 con 0x7fc2a80a98f0 2024-03-25T17:39:25.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.515+0000 7fc2a559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:25.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.515+0000 7fc2a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:25.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.515+0000 7fc2a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:25.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.515+0000 7fc2a559a640 10 monclient(hunting): my global_id is 4656 2024-03-25T17:39:25.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.515+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a80b28a0 con 0x7fc2a80a98f0 2024-03-25T17:39:25.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.517+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768275941 0 0) 0x7fc2a80b2a30 con 0x7fc2a813f710 2024-03-25T17:39:25.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.517+0000 7fc2a559a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:25.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.517+0000 7fc2a559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:25.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.517+0000 7fc2a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:25.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.517+0000 7fc2a559a640 10 monclient(hunting): my global_id is 34637 2024-03-25T17:39:25.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.517+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a8142520 con 0x7fc2a813f710 2024-03-25T17:39:25.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2117886878 0 0) 0x7fc290003520 con 0x7fc2a81430c0 2024-03-25T17:39:25.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 >> v1:172.21.15.134:6790/0 conn(0x7fc2a813f710 legacy=0x7fc2a81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:25.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 >> v1:172.21.15.134:6789/0 conn(0x7fc2a80a98f0 legacy=0x7fc2a80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 10 monclient: _finish_hunting 0 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 1 monclient: found mon.noname-c 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a80b44c0 con 0x7fc2a81430c0 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 10 monclient: _finish_auth 0 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 10 monclient: _check_auth_tickets 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc2a0001a10 con 0x7fc2a81430c0 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2a559a640 10 monclient: dump: 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:25.520 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2ac8c1640 5 monclient: authenticate success, global_id 4684 2024-03-25T17:39:25.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.518+0000 7fc2ac8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2a0001b70 con 0x7fc2a81430c0 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2a559a640 1 -- 172.21.15.134:0/3242973804 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc2a0001f80 con 0x7fc2a81430c0 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2ac8c1640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2ac8c1640 4 set_mon_vals no callback set 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.526+0000 7fc2a559a640 10 monclient: dump: 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:25.527 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.528 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.528 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:25.528 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:25.528 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:25.528 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:25.528 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:25.528 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:25.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.527+0000 7fc2ac8c1640 10 monclient: shutdown 2024-03-25T17:39:25.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.527+0000 7fc2ac8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:25.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.527+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/3242973804 >> v1:172.21.15.159:6789/0 conn(0x7fc2a81430c0 legacy=0x7fc2a814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:25.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.527+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/3242973804 shutdown_connections 2024-03-25T17:39:25.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.527+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/3242973804 >> 172.21.15.134:0/3242973804 conn(0x7fc2a80b7ea0 msgr2=0x7fc2a80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:25.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.527+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/3242973804 shutdown_connections 2024-03-25T17:39:25.529 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.528+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/3242973804 wait complete. 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 10 monclient: build_initial_monmap 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 10 monclient: monmap: 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:25.531 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 1 librados: starting msgr at 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 1 librados: starting objecter 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 1 Processor -- start 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 1 -- start start 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 1 librados: setting wanted keys 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 1 librados: calling monclient init 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.529+0000 7fc2ac8c1640 10 monclient: init 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 10 monclient(hunting): picked mon.noname-b con 0x7fc2a80a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 10 monclient(hunting): picked mon.noname-c con 0x7fc2a813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:25.532 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 10 monclient(hunting): picked mon.noname-a con 0x7fc2a81430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:25.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a80abfc0 con 0x7fc2a81430c0 2024-03-25T17:39:25.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a81dc150 con 0x7fc2a80a98f0 2024-03-25T17:39:25.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a8137980 con 0x7fc2a813f710 2024-03-25T17:39:25.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:25.533 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.530+0000 7fc2ac8c1640 10 monclient(hunting): authenticate will time out at 5918.793136s 2024-03-25T17:39:25.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.534+0000 7fc2a6d9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fc2a81430c0 0x7fc2a81dba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:56416/0 (socket says 172.21.15.134:56416) 2024-03-25T17:39:25.534 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.534+0000 7fc2a6d9d640 1 -- 172.21.15.134:0/40912244 learned_addr learned my addr 172.21.15.134:0/40912244 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:25.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.534+0000 7fc296ffd640 10 client.?.objecter ms_handle_connect 0x7fc2a81430c0 2024-03-25T17:39:25.535 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.534+0000 7fc296ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:25.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.538+0000 7fc296ffd640 10 client.?.objecter ms_handle_connect 0x7fc2a80a98f0 2024-03-25T17:39:25.538 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.538+0000 7fc296ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:25.539 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.539+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857592999 0 0) 0x7fc2a80abfc0 con 0x7fc2a81430c0 2024-03-25T17:39:25.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.539+0000 7fc296ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:25.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.539+0000 7fc296ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:25.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.539+0000 7fc296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:25.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.539+0000 7fc296ffd640 10 monclient(hunting): my global_id is 4659 2024-03-25T17:39:25.540 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.539+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc288004eb0 con 0x7fc2a81430c0 2024-03-25T17:39:25.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.548+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864740874 0 0) 0x7fc288004eb0 con 0x7fc2a81430c0 2024-03-25T17:39:25.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.548+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2a80abfc0 con 0x7fc2a81430c0 2024-03-25T17:39:25.548 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.548+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc2a0001ec0 con 0x7fc2a81430c0 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.548+0000 7fc296ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.548+0000 7fc296ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.548+0000 7fc296ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:25.549 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.548+0000 7fc296ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:25.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.550+0000 7fc296ffd640 10 client.?.objecter ms_handle_connect 0x7fc2a813f710 2024-03-25T17:39:25.551 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.550+0000 7fc296ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:25.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.553+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180831094 0 0) 0x7fc2a81dc150 con 0x7fc2a80a98f0 2024-03-25T17:39:25.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.553+0000 7fc296ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:25.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.553+0000 7fc296ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:25.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.553+0000 7fc296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:25.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.553+0000 7fc296ffd640 10 monclient(hunting): my global_id is 4687 2024-03-25T17:39:25.554 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.553+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc288004eb0 con 0x7fc2a80a98f0 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3170841931 0 0) 0x7fc2a80abfc0 con 0x7fc2a81430c0 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 >> v1:172.21.15.134:6790/0 conn(0x7fc2a813f710 legacy=0x7fc2a81db330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 >> v1:172.21.15.159:6789/0 conn(0x7fc2a80a98f0 legacy=0x7fc2a80ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 1 monclient: found mon.a 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a8139360 con 0x7fc2a81430c0 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:25.557 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 5 monclient: authenticate success, global_id 4659 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 10 monclient: _renew_subs 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a8139640 con 0x7fc2a81430c0 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 10 monclient: _renew_subs 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a8136f70 con 0x7fc2a81430c0 2024-03-25T17:39:25.558 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.556+0000 7fc2ac8c1640 1 librados: init done 2024-03-25T17:39:25.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.557+0000 7fc2ac8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:25.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.557+0000 7fc2ac8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:25.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.557+0000 7fc2ac8c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:25.559 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.557+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a8137b10 con 0x7fc2a81430c0 2024-03-25T17:39:25.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc2a0004e10 con 0x7fc2a81430c0 2024-03-25T17:39:25.883 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc296ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fc2a00015d0 con 0x7fc2a81430c0 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc296ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc296ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc296ffd640 10 monclient: dump: 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:25.884 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc2a4d99640 4 set_mon_vals no callback set 2024-03-25T17:39:25.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.883+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fc2a001afd0 con 0x7fc2a81430c0 2024-03-25T17:39:25.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.884+0000 7fc2a659c640 10 monclient: get_auth_request con 0x7fc28806dd90 auth_method 0 2024-03-25T17:39:25.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.884+0000 7fc296ffd640 10 client.4659.objecter ms_handle_connect 0x7fc28806dd90 2024-03-25T17:39:25.890 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1550706091 0 0) 0x7fc2a0054b80 con 0x7fc2a81430c0 2024-03-25T17:39:25.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 10 client.4659.objecter ms_dispatch 0x7fc2a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T17:39:25.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T17:39:25.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T17:39:25.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T17:39:25.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc2a00504c0 con 0x7fc2a81430c0 2024-03-25T17:39:25.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:25.891 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:25.890+0000 7fc296ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:26.114 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.114+0000 7fc2ac8c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:26.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.114+0000 7fc2ac8c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:26.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.114+0000 7fc2ac8c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:26.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.114+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2a80627b0 con 0x7fc2a81430c0 2024-03-25T17:39:26.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.180+0000 7fc296ffd640 1 -- 172.21.15.134:0/40912244 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7086 (unknown 238102980 0 4231624974) 0x7fc2a00525c0 con 0x7fc2a81430c0 2024-03-25T17:39:26.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.181+0000 7fc296ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:26.181 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.181+0000 7fc296ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:26.181 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:26.182 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":11,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:25.866846+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:25.866846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:26.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.182+0000 7fc2ac8c1640 10 librados: watch_flush enter 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 10 librados: watch_flush exit 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 10 client.4659.objecter successfully canceled tick 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 >> v1:172.21.15.134:6800/610837766 conn(0x7fc28806dd90 legacy=0x7fc288070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 10 monclient: shutdown 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 >> v1:172.21.15.134:6789/0 conn(0x7fc2a81430c0 legacy=0x7fc2a81dba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 shutdown_connections 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 >> 172.21.15.134:0/40912244 conn(0x7fc2a80b7ea0 msgr2=0x7fc2a8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 shutdown_connections 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 1 -- 172.21.15.134:0/40912244 wait complete. 2024-03-25T17:39:26.184 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:26.183+0000 7fc2ac8c1640 1 librados: shutdown 2024-03-25T17:39:26.194 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:32.999 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:33.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.112+0000 7f4618a4a640 10 monclient: get_monmap_and_config 2024-03-25T17:39:33.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.112+0000 7f4618a4a640 10 monclient: build_initial_monmap 2024-03-25T17:39:33.113 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.113+0000 7f4618a4a640 10 monclient: monmap: 2024-03-25T17:39:33.113 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:33.114002+0000 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:33.114002+0000 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:33.114 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:33.115 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.113+0000 7f4618a4a640 1 Processor -- start 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.113+0000 7f4618a4a640 1 -- start start 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.113+0000 7f4618a4a640 10 monclient: init 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f46140a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f461413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f46141430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4614142520 con 0x7f46140a98f0 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46140b28a0 con 0x7f461413f710 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46140b2a30 con 0x7f46141430c0 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:33.116 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.114+0000 7f4618a4a640 10 monclient(hunting): authenticate will time out at 5926.377198s 2024-03-25T17:39:33.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.126+0000 7f461259c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f46140a98f0 0x7f46140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34252/0 (socket says 172.21.15.134:34252) 2024-03-25T17:39:33.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.126+0000 7f461259c640 1 -- 172.21.15.134:0/400811198 learned_addr learned my addr 172.21.15.134:0/400811198 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:33.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.135+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025546217 0 0) 0x7f4614142520 con 0x7f46140a98f0 2024-03-25T17:39:33.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.135+0000 7f4611d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:33.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.135+0000 7f4611d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:33.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.135+0000 7f4611d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:33.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.135+0000 7f4611d9b640 10 monclient(hunting): my global_id is 4662 2024-03-25T17:39:33.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.135+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4600003520 con 0x7f46140a98f0 2024-03-25T17:39:33.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.136+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244170569 0 0) 0x7f46140b28a0 con 0x7f461413f710 2024-03-25T17:39:33.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.136+0000 7f4611d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:39:33.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.136+0000 7f4611d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:33.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.136+0000 7f4611d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:33.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.136+0000 7f4611d9b640 10 monclient(hunting): my global_id is 4690 2024-03-25T17:39:33.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.136+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4614142520 con 0x7f461413f710 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3073044031 0 0) 0x7f4614142520 con 0x7f461413f710 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 >> v1:172.21.15.134:6790/0 conn(0x7f46141430c0 legacy=0x7f461414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 >> v1:172.21.15.134:6789/0 conn(0x7f46140a98f0 legacy=0x7f46140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 1 monclient: found mon.noname-c 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:39:33.146 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46140b44c0 con 0x7f461413f710 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 10 monclient: _finish_auth 0 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f45fc001670 con 0x7f461413f710 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4611d9b640 10 monclient: dump: 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:33.147 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:33.148 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:33.148 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:33.148 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:33.148 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:33.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4618a4a640 5 monclient: authenticate success, global_id 4690 2024-03-25T17:39:33.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.145+0000 7f4618a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:33.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45fc0017d0 con 0x7f461413f710 2024-03-25T17:39:33.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4611d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:33.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4611d9b640 1 -- 172.21.15.134:0/400811198 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f45fc001b60 con 0x7f461413f710 2024-03-25T17:39:33.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4611d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:33.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4611d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:33.152 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4611d9b640 10 monclient: dump: 2024-03-25T17:39:33.152 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4618a4a640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:33.153 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.151+0000 7f4618a4a640 4 set_mon_vals no callback set 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.152+0000 7f4618a4a640 10 monclient: shutdown 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.152+0000 7f4618a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.152+0000 7f4618a4a640 1 -- 172.21.15.134:0/400811198 >> v1:172.21.15.159:6789/0 conn(0x7f461413f710 legacy=0x7f46141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.152+0000 7f4618a4a640 1 -- 172.21.15.134:0/400811198 shutdown_connections 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.152+0000 7f4618a4a640 1 -- 172.21.15.134:0/400811198 >> 172.21.15.134:0/400811198 conn(0x7f46140b7ea0 msgr2=0x7f46140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.152+0000 7f4618a4a640 1 -- 172.21.15.134:0/400811198 shutdown_connections 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.152+0000 7f4618a4a640 1 -- 172.21.15.134:0/400811198 wait complete. 2024-03-25T17:39:33.155 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.153+0000 7f4618a4a640 10 monclient: build_initial_monmap 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.153+0000 7f4618a4a640 10 monclient: monmap: 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.153+0000 7f4618a4a640 1 librados: starting msgr at 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.153+0000 7f4618a4a640 1 librados: starting objecter 2024-03-25T17:39:33.156 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 1 Processor -- start 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 1 -- start start 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 1 librados: setting wanted keys 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 1 librados: calling monclient init 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient: init 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f46140a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f461413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f46141430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46140abae0 con 0x7f461413f710 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46140acb60 con 0x7f46140a98f0 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46140abd50 con 0x7f46141430c0 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:33.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.154+0000 7f4618a4a640 10 monclient(hunting): authenticate will time out at 5926.417711s 2024-03-25T17:39:33.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.162+0000 7f4609d9b640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f461413f710 0x7f46140ac450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:34282/0 (socket says 172.21.15.134:34282) 2024-03-25T17:39:33.162 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.162+0000 7f4609d9b640 1 -- 172.21.15.134:0/1821311033 learned_addr learned my addr 172.21.15.134:0/1821311033 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:33.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.166+0000 7f460b7fe640 10 client.?.objecter ms_handle_connect 0x7f461413f710 2024-03-25T17:39:33.166 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.166+0000 7f460b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:33.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.172+0000 7f460b7fe640 10 client.?.objecter ms_handle_connect 0x7f46140a98f0 2024-03-25T17:39:33.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.172+0000 7f460b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454778004 0 0) 0x7f46140abae0 con 0x7f461413f710 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 10 monclient(hunting): my global_id is 4665 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f4004eb0 con 0x7f461413f710 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 10 client.?.objecter ms_handle_connect 0x7f46141430c0 2024-03-25T17:39:33.183 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.182+0000 7f460b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:33.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.190+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2990529825 0 0) 0x7f46140acb60 con 0x7f46140a98f0 2024-03-25T17:39:33.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.190+0000 7f460b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:33.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.190+0000 7f460b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:33.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.190+0000 7f460b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:33.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.190+0000 7f460b7fe640 10 monclient(hunting): my global_id is 4693 2024-03-25T17:39:33.191 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.190+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46140abae0 con 0x7f46140a98f0 2024-03-25T17:39:33.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.193+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819400868 0 0) 0x7f46140abd50 con 0x7f46141430c0 2024-03-25T17:39:33.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.193+0000 7f460b7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:39:33.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.193+0000 7f460b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:33.193 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.193+0000 7f460b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:33.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.193+0000 7f460b7fe640 10 monclient(hunting): my global_id is 34646 2024-03-25T17:39:33.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.193+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46140acb60 con 0x7f46141430c0 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.195+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991088658 0 0) 0x7f45f4004eb0 con 0x7f461413f710 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.195+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46140abd50 con 0x7f461413f710 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.195+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f45fc001ba0 con 0x7f461413f710 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.195+0000 7f460b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.195+0000 7f460b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.195+0000 7f460b7fe640 10 monclient(hunting): dump: 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:33.196 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:33.197 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:33.197 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:33.197 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:33.197 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:33.197 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.195+0000 7f460b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:33.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 468250020 0 0) 0x7f46140abd50 con 0x7f461413f710 2024-03-25T17:39:33.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 >> v1:172.21.15.134:6790/0 conn(0x7f46141430c0 legacy=0x7f46141e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:33.202 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 >> v1:172.21.15.159:6789/0 conn(0x7f46140a98f0 legacy=0x7f4614138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 1 monclient: found mon.a 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46140ac030 con 0x7f461413f710 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 10 monclient: _finish_auth 0 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f460b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.201+0000 7f4618a4a640 5 monclient: authenticate success, global_id 4665 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 10 monclient: _renew_subs 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46141390d0 con 0x7f461413f710 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 10 client.4665.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:33.203 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 10 monclient: _renew_subs 2024-03-25T17:39:33.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:33.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4614136520 con 0x7f461413f710 2024-03-25T17:39:33.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.202+0000 7f4618a4a640 1 librados: init done 2024-03-25T17:39:33.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.203+0000 7f4618a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:33.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.203+0000 7f4618a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:33.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.203+0000 7f4618a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:33.204 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.203+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46140627b0 con 0x7f461413f710 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45fc0038a0 con 0x7f461413f710 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f460b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f45fc003cb0 con 0x7f461413f710 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f460b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f460b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f460b7fe640 10 monclient: dump: 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:33.507 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f461159a640 4 set_mon_vals no callback set 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.506+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f45fc003ed0 con 0x7f461413f710 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.507+0000 7f461259c640 10 monclient: get_auth_request con 0x7f45f406ddf0 auth_method 0 2024-03-25T17:39:33.508 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.507+0000 7f460b7fe640 10 client.4665.objecter ms_handle_connect 0x7f45f406ddf0 2024-03-25T17:39:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f45fc0549b0 con 0x7f461413f710 2024-03-25T17:39:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 10 client.4665.objecter ms_dispatch 0x7f4614000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:39:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 3 client.4665.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:39:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 3 client.4665.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:39:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 20 client.4665.objecter dump_active .. 0 homeless 2024-03-25T17:39:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f45fc0504c0 con 0x7f461413f710 2024-03-25T17:39:33.513 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:33.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.512+0000 7f460b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:33.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.768+0000 7f4618a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:33.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.768+0000 7f4618a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:33.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.768+0000 7f4618a4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:33.769 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.768+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f461413ce70 con 0x7f461413f710 2024-03-25T17:39:33.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.826+0000 7f460b7fe640 1 -- 172.21.15.134:0/1821311033 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f45fc0523f0 con 0x7f461413f710 2024-03-25T17:39:33.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.826+0000 7f460b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:33.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.826+0000 7f460b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:33.827 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:33.827 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 10 librados: watch_flush enter 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 10 librados: watch_flush exit 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 20 client.4665.objecter shutdown clearing up homeless session... 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 10 client.4665.objecter successfully canceled tick 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 >> v1:172.21.15.134:6800/610837766 conn(0x7f45f406ddf0 legacy=0x7f45f40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 10 monclient: shutdown 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 >> v1:172.21.15.134:6789/0 conn(0x7f461413f710 legacy=0x7f46140ac450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 shutdown_connections 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 >> 172.21.15.134:0/1821311033 conn(0x7f46140b7ea0 msgr2=0x7f4614074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:33.830 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 shutdown_connections 2024-03-25T17:39:33.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 1 -- 172.21.15.134:0/1821311033 wait complete. 2024-03-25T17:39:33.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:33.829+0000 7f4618a4a640 1 librados: shutdown 2024-03-25T17:39:33.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:36.051 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:41.067 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.059 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:41.075 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.092 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:10:41.109 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.095 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:41.112 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.107 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:41.136 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.111 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:41.140 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.115 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:41.132 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.142 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:10:41.158 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:36.153 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:10:41.182 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:40.648 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:40.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.758+0000 7f9758c25640 10 monclient: get_monmap_and_config 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.758+0000 7f9758c25640 10 monclient: build_initial_monmap 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.759+0000 7f9758c25640 10 monclient: monmap: 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:40.761000+0000 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:40.761000+0000 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:40.761 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.759+0000 7f9758c25640 1 Processor -- start 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.759+0000 7f9758c25640 1 -- start start 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.759+0000 7f9758c25640 10 monclient: init 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 10 monclient(hunting): picked mon.noname-c con 0x7f97540a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 10 monclient(hunting): picked mon.noname-a con 0x7f975413f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 10 monclient(hunting): picked mon.noname-b con 0x7f97541430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9754142520 con 0x7f975413f710 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540b28a0 con 0x7f97540a98f0 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97540b2a30 con 0x7f97541430c0 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:40.763 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.760+0000 7f9758c25640 10 monclient(hunting): authenticate will time out at 5934.024669s 2024-03-25T17:39:40.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.766+0000 7f9752d9d640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f97541430c0 0x7f975414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46594/0 (socket says 172.21.15.134:46594) 2024-03-25T17:39:40.768 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.766+0000 7f9752d9d640 1 -- 172.21.15.134:0/1427160298 learned_addr learned my addr 172.21.15.134:0/1427160298 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:40.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.777+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3121290625 0 0) 0x7f9754142520 con 0x7f975413f710 2024-03-25T17:39:40.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.777+0000 7f975159a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:40.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.777+0000 7f975159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:40.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.777+0000 7f975159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:40.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.777+0000 7f975159a640 10 monclient(hunting): my global_id is 4668 2024-03-25T17:39:40.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.777+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c003520 con 0x7f975413f710 2024-03-25T17:39:40.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.783+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473802216 0 0) 0x7f97540b28a0 con 0x7f97540a98f0 2024-03-25T17:39:40.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.783+0000 7f975159a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:40.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.783+0000 7f975159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:40.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.783+0000 7f975159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:40.785 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.783+0000 7f975159a640 10 monclient(hunting): my global_id is 4696 2024-03-25T17:39:40.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9754142520 con 0x7f97540a98f0 2024-03-25T17:39:40.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 325449113 0 0) 0x7f973c003520 con 0x7f975413f710 2024-03-25T17:39:40.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 >> v1:172.21.15.134:6790/0 conn(0x7f97541430c0 legacy=0x7f975414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 >> v1:172.21.15.159:6789/0 conn(0x7f97540a98f0 legacy=0x7f97540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 10 monclient: _finish_hunting 0 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 1 monclient: found mon.noname-a 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97540b44c0 con 0x7f975413f710 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 10 monclient: _finish_auth 0 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 10 monclient: _check_auth_tickets 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f975159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f9758c25640 5 monclient: authenticate success, global_id 4668 2024-03-25T17:39:40.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.784+0000 7f9758c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.788+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9748001650 con 0x7f975413f710 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.788+0000 7f975159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.788+0000 7f975159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.788+0000 7f975159a640 10 monclient: dump: 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:40.790 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:40.791 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:40.791 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.788+0000 7f9758c25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97480018a0 con 0x7f975413f710 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f975159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f975159a640 1 -- 172.21.15.134:0/1427160298 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9748001c30 con 0x7f975413f710 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f975159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f975159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f975159a640 10 monclient: dump: 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.801 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f9758c25640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:40.802 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.799+0000 7f9758c25640 4 set_mon_vals no callback set 2024-03-25T17:39:40.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.800+0000 7f9758c25640 10 monclient: shutdown 2024-03-25T17:39:40.803 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.800+0000 7f9758c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.800+0000 7f9758c25640 1 -- 172.21.15.134:0/1427160298 >> v1:172.21.15.134:6789/0 conn(0x7f975413f710 legacy=0x7f97541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.800+0000 7f9758c25640 1 -- 172.21.15.134:0/1427160298 shutdown_connections 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.800+0000 7f9758c25640 1 -- 172.21.15.134:0/1427160298 >> 172.21.15.134:0/1427160298 conn(0x7f97540b7ea0 msgr2=0x7f97540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.800+0000 7f9758c25640 1 -- 172.21.15.134:0/1427160298 shutdown_connections 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.800+0000 7f9758c25640 1 -- 172.21.15.134:0/1427160298 wait complete. 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.801+0000 7f9758c25640 10 monclient: build_initial_monmap 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.801+0000 7f9758c25640 10 monclient: monmap: 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:40.804 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.802+0000 7f9758c25640 1 librados: starting msgr at 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.802+0000 7f9758c25640 1 librados: starting objecter 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.802+0000 7f9758c25640 1 Processor -- start 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.802+0000 7f9758c25640 1 -- start start 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.802+0000 7f9758c25640 1 librados: setting wanted keys 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.802+0000 7f9758c25640 1 librados: calling monclient init 2024-03-25T17:39:40.805 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.802+0000 7f9758c25640 10 monclient: init 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 10 monclient(hunting): picked mon.noname-a con 0x7f97540a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 10 monclient(hunting): picked mon.noname-b con 0x7f975413f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 10 monclient(hunting): picked mon.noname-c con 0x7f97541430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9754137260 con 0x7f97540a98f0 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97541373f0 con 0x7f975413f710 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9754137600 con 0x7f97541430c0 2024-03-25T17:39:40.806 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:40.807 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.803+0000 7f9758c25640 10 monclient(hunting): authenticate will time out at 5934.067007s 2024-03-25T17:39:40.811 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.810+0000 7f9751d9b640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f975413f710 0x7f9754133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:36386/0 (socket says 172.21.15.134:36386) 2024-03-25T17:39:40.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.810+0000 7f9751d9b640 1 -- 172.21.15.134:0/1246437121 learned_addr learned my addr 172.21.15.134:0/1246437121 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:40.812 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.811+0000 7f9742ffd640 10 client.?.objecter ms_handle_connect 0x7f975413f710 2024-03-25T17:39:40.813 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.811+0000 7f9742ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:40.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.822+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229095869 0 0) 0x7f97540b44c0 con 0x7f975413f710 2024-03-25T17:39:40.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.823+0000 7f9742ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:40.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.823+0000 7f9742ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:40.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.823+0000 7f9742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:40.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.823+0000 7f9742ffd640 10 monclient(hunting): my global_id is 4699 2024-03-25T17:39:40.825 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.823+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9734004eb0 con 0x7f975413f710 2024-03-25T17:39:40.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.826+0000 7f9742ffd640 10 client.?.objecter ms_handle_connect 0x7f97540a98f0 2024-03-25T17:39:40.828 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.826+0000 7f9742ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:40.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.828+0000 7f9742ffd640 10 client.?.objecter ms_handle_connect 0x7f97541430c0 2024-03-25T17:39:40.829 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.828+0000 7f9742ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:40.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.830+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2749196480 0 0) 0x7f9754137260 con 0x7f97540a98f0 2024-03-25T17:39:40.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.830+0000 7f9742ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:40.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.830+0000 7f9742ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:40.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.830+0000 7f9742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:40.831 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.830+0000 7f9742ffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T17:39:40.832 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.830+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97540b44c0 con 0x7f97540a98f0 2024-03-25T17:39:40.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.837+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1060724475 0 0) 0x7f9734004eb0 con 0x7f975413f710 2024-03-25T17:39:40.838 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.837+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9754137260 con 0x7f975413f710 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.837+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9748001ad0 con 0x7f975413f710 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.837+0000 7f9742ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.837+0000 7f9742ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.837+0000 7f9742ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:40.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.837+0000 7f9742ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:40.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.843+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459503788 0 0) 0x7f9754137600 con 0x7f97541430c0 2024-03-25T17:39:40.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.843+0000 7f9742ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:40.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.843+0000 7f9742ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:40.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.843+0000 7f9742ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:40.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.843+0000 7f9742ffd640 10 monclient(hunting): my global_id is 34652 2024-03-25T17:39:40.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.843+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9734004eb0 con 0x7f97541430c0 2024-03-25T17:39:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.848+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563684420 0 0) 0x7f97540b44c0 con 0x7f97540a98f0 2024-03-25T17:39:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.849+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9754137600 con 0x7f97540a98f0 2024-03-25T17:39:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.849+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9744001930 con 0x7f97540a98f0 2024-03-25T17:39:40.850 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.849+0000 7f9742ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.849+0000 7f9742ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.849+0000 7f9742ffd640 10 monclient(hunting): dump: 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:40.851 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3390825922 0 0) 0x7f9754137260 con 0x7f975413f710 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 >> v1:172.21.15.134:6790/0 conn(0x7f97541430c0 legacy=0x7f97541e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 >> v1:172.21.15.134:6789/0 conn(0x7f97540a98f0 legacy=0x7f97541d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 1 monclient: found mon.b 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9754137790 con 0x7f975413f710 2024-03-25T17:39:40.852 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 10 monclient: _finish_auth 0 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9742ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 5 monclient: authenticate success, global_id 4699 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 10 monclient: _renew_subs 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9754137f00 con 0x7f975413f710 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 10 client.4699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 10 monclient: _renew_subs 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97541d7ef0 con 0x7f975413f710 2024-03-25T17:39:40.853 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.850+0000 7f9758c25640 1 librados: init done 2024-03-25T17:39:40.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.851+0000 7f9758c25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:40.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.851+0000 7f9758c25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:40.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.851+0000 7f9758c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:40.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:40.851+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9754137ba0 con 0x7f975413f710 2024-03-25T17:39:41.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.063+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9748001230 con 0x7f975413f710 2024-03-25T17:39:41.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.063+0000 7f9742ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:41.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.064+0000 7f9750d99640 4 set_mon_vals no callback set 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.122+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f9748003b00 con 0x7f975413f710 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.122+0000 7f9742ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.122+0000 7f9742ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.122+0000 7f9742ffd640 10 monclient: dump: 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:41.124 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:41.125 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:41.125 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:41.125 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:41.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.123+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f9748003ce0 con 0x7f975413f710 2024-03-25T17:39:41.125 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.124+0000 7f975259c640 10 monclient: get_auth_request con 0x7f973406df60 auth_method 0 2024-03-25T17:39:41.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.124+0000 7f9742ffd640 10 client.4699.objecter ms_handle_connect 0x7f973406df60 2024-03-25T17:39:41.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.126+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f9748054810 con 0x7f975413f710 2024-03-25T17:39:41.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.126+0000 7f9742ffd640 10 client.4699.objecter ms_dispatch 0x7f9754000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:39:41.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.126+0000 7f9742ffd640 3 client.4699.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:39:41.127 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.126+0000 7f9742ffd640 3 client.4699.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:39:41.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.126+0000 7f9742ffd640 20 client.4699.objecter dump_active .. 0 homeless 2024-03-25T17:39:41.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.129+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9748050320 con 0x7f975413f710 2024-03-25T17:39:41.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.129+0000 7f9742ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:41.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.129+0000 7f9742ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:41.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.352+0000 7f9758c25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:41.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.352+0000 7f9758c25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:41.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.352+0000 7f9758c25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:41.354 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.352+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9754074cc0 con 0x7f975413f710 2024-03-25T17:39:41.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.455+0000 7f9742ffd640 1 -- 172.21.15.134:0/1246437121 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f9748052250 con 0x7f975413f710 2024-03-25T17:39:41.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.455+0000 7f9742ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:41.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.455+0000 7f9742ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:41.456 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:41.457 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:41.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 10 librados: watch_flush enter 2024-03-25T17:39:41.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 10 librados: watch_flush exit 2024-03-25T17:39:41.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 20 client.4699.objecter shutdown clearing up homeless session... 2024-03-25T17:39:41.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 10 client.4699.objecter successfully canceled tick 2024-03-25T17:39:41.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 >> v1:172.21.15.134:6800/610837766 conn(0x7f973406df60 legacy=0x7f9734070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:41.459 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 10 monclient: shutdown 2024-03-25T17:39:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 >> v1:172.21.15.159:6789/0 conn(0x7f975413f710 legacy=0x7f9754133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 shutdown_connections 2024-03-25T17:39:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 >> 172.21.15.134:0/1246437121 conn(0x7f97540b7ea0 msgr2=0x7f9754074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 shutdown_connections 2024-03-25T17:39:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 1 -- 172.21.15.134:0/1246437121 wait complete. 2024-03-25T17:39:41.460 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:41.457+0000 7f9758c25640 1 librados: shutdown 2024-03-25T17:39:41.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:48.279 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.395+0000 7fdbb0c55640 10 monclient: get_monmap_and_config 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.395+0000 7fdbb0c55640 10 monclient: build_initial_monmap 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.395+0000 7fdbb0c55640 10 monclient: monmap: 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:48.396497+0000 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:48.396497+0000 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:48.396 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:48.397 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:48.397 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:48.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.396+0000 7fdbb0c55640 1 Processor -- start 2024-03-25T17:39:48.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.396+0000 7fdbb0c55640 1 -- start start 2024-03-25T17:39:48.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.396+0000 7fdbb0c55640 10 monclient: init 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 10 monclient(hunting): picked mon.noname-a con 0x7fdbac06ab80 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 10 monclient(hunting): picked mon.noname-b con 0x7fdbac06af60 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 10 monclient(hunting): picked mon.noname-c con 0x7fdbac0b00b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac0b0500 con 0x7fdbac06ab80 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac13ffa0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac0af520 con 0x7fdbac06af60 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:48.398 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.397+0000 7fdbb0c55640 10 monclient(hunting): authenticate will time out at 5941.660149s 2024-03-25T17:39:48.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.404+0000 7fdbaa59c640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7fdbac06ab80 0x7fdbac0ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:36656/0 (socket says 172.21.15.134:36656) 2024-03-25T17:39:48.404 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.404+0000 7fdbaa59c640 1 -- 172.21.15.134:0/2710795083 learned_addr learned my addr 172.21.15.134:0/2710795083 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:48.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.410+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180392990 0 0) 0x7fdbac0b0500 con 0x7fdbac06ab80 2024-03-25T17:39:48.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.410+0000 7fdba959a640 10 monclient(hunting): _negotiate 2024-03-25T17:39:48.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.410+0000 7fdba959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:48.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.410+0000 7fdba959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:48.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.410+0000 7fdba959a640 10 monclient(hunting): my global_id is 4674 2024-03-25T17:39:48.411 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.410+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb94003520 con 0x7fdbac06ab80 2024-03-25T17:39:48.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289932010 0 0) 0x7fdb94003520 con 0x7fdbac06ab80 2024-03-25T17:39:48.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 >> v1:172.21.15.134:6790/0 conn(0x7fdbac06af60 legacy=0x7fdbac13f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:48.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 >> v1:172.21.15.159:6789/0 conn(0x7fdbac0b00b0 legacy=0x7fdbac14dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:48.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 10 monclient: _finish_hunting 0 2024-03-25T17:39:48.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 1 monclient: found mon.noname-a 2024-03-25T17:39:48.421 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac072890 con 0x7fdbac06ab80 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 10 monclient: _finish_auth 0 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 10 monclient: _check_auth_tickets 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fdb9c001690 con 0x7fdbac06ab80 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdbb0c55640 5 monclient: authenticate success, global_id 4674 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdbb0c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdba959a640 10 monclient: dump: 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.422 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.423 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:48.423 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:48.423 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:48.423 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:48.423 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:48.423 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:48.423 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.420+0000 7fdbb0c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:48.429 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb9c0017f0 con 0x7fdbac06ab80 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdba959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdba959a640 1 -- 172.21.15.134:0/2710795083 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fdb9c001c00 con 0x7fdbac06ab80 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdbb0c55640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdbb0c55640 4 set_mon_vals no callback set 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdba959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdba959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.429+0000 7fdba959a640 10 monclient: dump: 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:48.430 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:48.431 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:48.431 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:48.431 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:48.431 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.430+0000 7fdbb0c55640 10 monclient: shutdown 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.430+0000 7fdbb0c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.430+0000 7fdbb0c55640 1 -- 172.21.15.134:0/2710795083 >> v1:172.21.15.134:6789/0 conn(0x7fdbac06ab80 legacy=0x7fdbac0ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.430+0000 7fdbb0c55640 1 -- 172.21.15.134:0/2710795083 shutdown_connections 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.430+0000 7fdbb0c55640 1 -- 172.21.15.134:0/2710795083 >> 172.21.15.134:0/2710795083 conn(0x7fdbac0b4a80 msgr2=0x7fdbac0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.430+0000 7fdbb0c55640 1 -- 172.21.15.134:0/2710795083 shutdown_connections 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.430+0000 7fdbb0c55640 1 -- 172.21.15.134:0/2710795083 wait complete. 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.431+0000 7fdbb0c55640 10 monclient: build_initial_monmap 2024-03-25T17:39:48.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.431+0000 7fdbb0c55640 10 monclient: monmap: 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.431+0000 7fdbb0c55640 1 librados: starting msgr at 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.431+0000 7fdbb0c55640 1 librados: starting objecter 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.431+0000 7fdbb0c55640 1 Processor -- start 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 1 -- start start 2024-03-25T17:39:48.433 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 1 librados: setting wanted keys 2024-03-25T17:39:48.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 1 librados: calling monclient init 2024-03-25T17:39:48.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient: init 2024-03-25T17:39:48.434 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient(hunting): picked mon.noname-c con 0x7fdbac06ab80 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient(hunting): picked mon.noname-b con 0x7fdbac0b00b0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient(hunting): picked mon.noname-a con 0x7fdbac1daac0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac1daf70 con 0x7fdbac1daac0 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac145450 con 0x7fdbac0b00b0 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac145cf0 con 0x7fdbac06ab80 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:48.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.432+0000 7fdbb0c55640 10 monclient(hunting): authenticate will time out at 5941.695376s 2024-03-25T17:39:48.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.439+0000 7fdbaa59c640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fdbac06ab80 0x7fdbac146210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:51218/0 (socket says 172.21.15.134:51218) 2024-03-25T17:39:48.439 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.439+0000 7fdbaa59c640 1 -- 172.21.15.134:0/59405783 learned_addr learned my addr 172.21.15.134:0/59405783 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:48.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.440+0000 7fdb9affd640 10 client.?.objecter ms_handle_connect 0x7fdbac06ab80 2024-03-25T17:39:48.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.440+0000 7fdb9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:48.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.444+0000 7fdb9affd640 10 client.?.objecter ms_handle_connect 0x7fdbac0b00b0 2024-03-25T17:39:48.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.444+0000 7fdb9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:48.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.448+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600202013 0 0) 0x7fdbac145cf0 con 0x7fdbac06ab80 2024-03-25T17:39:48.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.448+0000 7fdb9affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:48.448 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.448+0000 7fdb9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:48.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.448+0000 7fdb9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:48.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.448+0000 7fdb9affd640 10 monclient(hunting): my global_id is 34658 2024-03-25T17:39:48.449 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.448+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb8c004eb0 con 0x7fdbac06ab80 2024-03-25T17:39:48.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.455+0000 7fdb9affd640 10 client.?.objecter ms_handle_connect 0x7fdbac1daac0 2024-03-25T17:39:48.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.455+0000 7fdb9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:48.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.456+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616759726 0 0) 0x7fdbac145450 con 0x7fdbac0b00b0 2024-03-25T17:39:48.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.456+0000 7fdb9affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:48.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.456+0000 7fdb9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:48.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.456+0000 7fdb9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:48.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.456+0000 7fdb9affd640 10 monclient(hunting): my global_id is 4705 2024-03-25T17:39:48.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.456+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbac145cf0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.461+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4041354865 0 0) 0x7fdb8c004eb0 con 0x7fdbac06ab80 2024-03-25T17:39:48.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.461+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbac145450 con 0x7fdbac06ab80 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.461+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fdb9c001bc0 con 0x7fdbac06ab80 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.461+0000 7fdb9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.461+0000 7fdb9affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.461+0000 7fdb9affd640 10 monclient(hunting): dump: 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:48.462 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:48.463 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:48.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.461+0000 7fdb9affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:48.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.469+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4117073975 0 0) 0x7fdbac145cf0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.470+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb8c004eb0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.470+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fdba00018b0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.470+0000 7fdb9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.470+0000 7fdb9affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.470+0000 7fdb9affd640 10 monclient(hunting): dump: 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:48.471 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:48.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.480+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413492562 0 0) 0x7fdbac1daf70 con 0x7fdbac1daac0 2024-03-25T17:39:48.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.480+0000 7fdb9affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:48.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.480+0000 7fdb9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:48.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.480+0000 7fdb9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:48.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.480+0000 7fdb9affd640 10 monclient(hunting): my global_id is 4677 2024-03-25T17:39:48.481 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.480+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbac145cf0 con 0x7fdbac1daac0 2024-03-25T17:39:48.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2107254797 0 0) 0x7fdb8c004eb0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 >> v1:172.21.15.134:6790/0 conn(0x7fdbac06ab80 legacy=0x7fdbac146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:48.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 >> v1:172.21.15.134:6789/0 conn(0x7fdbac1daac0 legacy=0x7fdbac1436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:48.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:48.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 1 monclient: found mon.b 2024-03-25T17:39:48.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:48.492 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac1389d0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 10 monclient: _finish_auth 0 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdb9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 5 monclient: authenticate success, global_id 4705 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 10 monclient: _renew_subs 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbac1d4810 con 0x7fdbac0b00b0 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 10 monclient: _renew_subs 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbac1d4cf0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.491+0000 7fdbb0c55640 1 librados: init done 2024-03-25T17:39:48.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.492+0000 7fdbb0c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:48.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.492+0000 7fdbb0c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:48.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.492+0000 7fdbb0c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:48.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.492+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbac145e80 con 0x7fdbac0b00b0 2024-03-25T17:39:48.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.763+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdba0003610 con 0x7fdbac0b00b0 2024-03-25T17:39:48.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.763+0000 7fdb9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:48.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.763+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fdba0003a60 con 0x7fdbac0b00b0 2024-03-25T17:39:48.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.763+0000 7fdb9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:48.764 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.763+0000 7fdb9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.763+0000 7fdb9affd640 10 monclient: dump: 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:48.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.763+0000 7fdba8d99640 4 set_mon_vals no callback set 2024-03-25T17:39:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.764+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fdba0003c80 con 0x7fdbac0b00b0 2024-03-25T17:39:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.765+0000 7fdbaa59c640 10 monclient: get_auth_request con 0x7fdb8c06df30 auth_method 0 2024-03-25T17:39:48.766 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.765+0000 7fdb9affd640 10 client.4705.objecter ms_handle_connect 0x7fdb8c06df30 2024-03-25T17:39:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.770+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7fdba00535f0 con 0x7fdbac0b00b0 2024-03-25T17:39:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.770+0000 7fdb9affd640 10 client.4705.objecter ms_dispatch 0x7fdbac000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:39:48.770 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.770+0000 7fdb9affd640 3 client.4705.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:39:48.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.770+0000 7fdb9affd640 3 client.4705.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:39:48.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.770+0000 7fdb9affd640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T17:39:48.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.771+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdba004f100 con 0x7fdbac0b00b0 2024-03-25T17:39:48.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.771+0000 7fdb9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:48.771 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:48.771+0000 7fdb9affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:49.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.000+0000 7fdbb0c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:49.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.000+0000 7fdbb0c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:49.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.000+0000 7fdbb0c55640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:39:49.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.000+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbac0627b0 con 0x7fdbac0b00b0 2024-03-25T17:39:49.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.023+0000 7fdb9affd640 1 -- 172.21.15.134:0/59405783 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7fdba0051030 con 0x7fdbac0b00b0 2024-03-25T17:39:49.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.023+0000 7fdb9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:49.023 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.023+0000 7fdb9affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:49.024 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:49.024 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.025+0000 7fdbb0c55640 10 librados: watch_flush enter 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.025+0000 7fdbb0c55640 10 librados: watch_flush exit 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.025+0000 7fdbb0c55640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.025+0000 7fdbb0c55640 10 client.4705.objecter successfully canceled tick 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 >> v1:172.21.15.134:6800/610837766 conn(0x7fdb8c06df30 legacy=0x7fdb8c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 10 monclient: shutdown 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 >> v1:172.21.15.159:6789/0 conn(0x7fdbac0b00b0 legacy=0x7fdbac143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 shutdown_connections 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 >> 172.21.15.134:0/59405783 conn(0x7fdbac0b4a80 msgr2=0x7fdbac0ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 shutdown_connections 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 1 -- 172.21.15.134:0/59405783 wait complete. 2024-03-25T17:39:49.027 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:49.026+0000 7fdbb0c55640 1 librados: shutdown 2024-03-25T17:39:49.038 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:39:55.845 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:39:55.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.959+0000 7fef822c5640 10 monclient: get_monmap_and_config 2024-03-25T17:39:55.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.959+0000 7fef822c5640 10 monclient: build_initial_monmap 2024-03-25T17:39:55.960 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.959+0000 7fef822c5640 10 monclient: monmap: 2024-03-25T17:39:55.960 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:55.960 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:55.960 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:39:55.960652+0000 2024-03-25T17:39:55.960 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:39:55.960652+0000 2024-03-25T17:39:55.961 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:55.961 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:55.961 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:55.961 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:39:55.961 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:39:55.961 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.960+0000 7fef822c5640 1 Processor -- start 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.960+0000 7fef822c5640 1 -- start start 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.960+0000 7fef822c5640 10 monclient: init 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 10 monclient(hunting): picked mon.noname-b con 0x7fef7c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 10 monclient(hunting): picked mon.noname-c con 0x7fef7c13f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 10 monclient(hunting): picked mon.noname-a con 0x7fef7c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c142520 con 0x7fef7c1430c0 2024-03-25T17:39:55.962 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0b28a0 con 0x7fef7c13f710 2024-03-25T17:39:55.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c0b2a30 con 0x7fef7c0a98f0 2024-03-25T17:39:55.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:55.963 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.961+0000 7fef822c5640 10 monclient(hunting): authenticate will time out at 5949.224089s 2024-03-25T17:39:55.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.966+0000 7fef7b7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7fef7c13f710 0x7fef7c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:43328/0 (socket says 172.21.15.134:43328) 2024-03-25T17:39:55.967 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.966+0000 7fef7b7fe640 1 -- 172.21.15.134:0/2501537418 learned_addr learned my addr 172.21.15.134:0/2501537418 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:55.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.980+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356377511 0 0) 0x7fef7c0b28a0 con 0x7fef7c13f710 2024-03-25T17:39:55.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.980+0000 7fef7affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:55.980 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.980+0000 7fef7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:55.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.980+0000 7fef7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:55.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.980+0000 7fef7affd640 10 monclient(hunting): my global_id is 4708 2024-03-25T17:39:55.981 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.980+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef64003520 con 0x7fef7c13f710 2024-03-25T17:39:55.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.986+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383335749 0 0) 0x7fef7c142520 con 0x7fef7c1430c0 2024-03-25T17:39:55.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.986+0000 7fef7affd640 10 monclient(hunting): _negotiate 2024-03-25T17:39:55.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.986+0000 7fef7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:55.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.986+0000 7fef7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:55.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.986+0000 7fef7affd640 10 monclient(hunting): my global_id is 4680 2024-03-25T17:39:55.987 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.986+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef7c0b28a0 con 0x7fef7c1430c0 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2731690779 0 0) 0x7fef64003520 con 0x7fef7c13f710 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 >> v1:172.21.15.134:6790/0 conn(0x7fef7c0a98f0 legacy=0x7fef7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 >> v1:172.21.15.134:6789/0 conn(0x7fef7c1430c0 legacy=0x7fef7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 10 monclient: _finish_hunting 0 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 1 monclient: found mon.noname-c 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c0b44c0 con 0x7fef7c13f710 2024-03-25T17:39:56.000 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 10 monclient: _finish_auth 0 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 10 monclient: _check_auth_tickets 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fef70001690 con 0x7fef7c13f710 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef822c5640 5 monclient: authenticate success, global_id 4708 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef822c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef7affd640 10 monclient: dump: 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:56.001 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:56.002 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:56.002 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:56.002 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:56.002 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:56.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:55.999+0000 7fef822c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:39:56.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef700017f0 con 0x7fef7c13f710 2024-03-25T17:39:56.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:56.011 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef7affd640 1 -- 172.21.15.134:0/2501537418 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fef70001c00 con 0x7fef7c13f710 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef7affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef7affd640 10 monclient: dump: 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:56.012 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:56.013 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:56.013 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:56.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 10 monclient: get_monmap_and_config success 2024-03-25T17:39:56.013 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 4 set_mon_vals no callback set 2024-03-25T17:39:56.014 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 10 monclient: shutdown 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 1 -- 172.21.15.134:0/2501537418 >> v1:172.21.15.159:6789/0 conn(0x7fef7c13f710 legacy=0x7fef7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 1 -- 172.21.15.134:0/2501537418 shutdown_connections 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 1 -- 172.21.15.134:0/2501537418 >> 172.21.15.134:0/2501537418 conn(0x7fef7c0b7ea0 msgr2=0x7fef7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.011+0000 7fef822c5640 1 -- 172.21.15.134:0/2501537418 shutdown_connections 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.012+0000 7fef822c5640 1 -- 172.21.15.134:0/2501537418 wait complete. 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.012+0000 7fef822c5640 10 monclient: build_initial_monmap 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.012+0000 7fef822c5640 10 monclient: monmap: 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:39:56.015 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.013+0000 7fef822c5640 1 librados: starting msgr at 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.013+0000 7fef822c5640 1 librados: starting objecter 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.013+0000 7fef822c5640 1 Processor -- start 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.013+0000 7fef822c5640 1 -- start start 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.013+0000 7fef822c5640 1 librados: setting wanted keys 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.013+0000 7fef822c5640 1 librados: calling monclient init 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.013+0000 7fef822c5640 10 monclient: init 2024-03-25T17:39:56.016 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 10 monclient: _reopen_session rank -1 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 10 monclient(hunting): picked mon.noname-c con 0x7fef7c0a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 10 monclient(hunting): picked mon.noname-a con 0x7fef7c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 10 monclient(hunting): picked mon.noname-b con 0x7fef7c1430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c070210 con 0x7fef7c13f710 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c06dea0 con 0x7fef7c1430c0 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7c06e680 con 0x7fef7c0a98f0 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 10 monclient(hunting): _renew_subs 2024-03-25T17:39:56.017 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.014+0000 7fef822c5640 10 monclient(hunting): authenticate will time out at 5949.277141s 2024-03-25T17:39:56.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.021+0000 7fef7bfff640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7fef7c0a98f0 0x7fef7c1dbf10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:51972/0 (socket says 172.21.15.134:51972) 2024-03-25T17:39:56.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.021+0000 7fef7bfff640 1 -- 172.21.15.134:0/2576144731 learned_addr learned my addr 172.21.15.134:0/2576144731 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:39:56.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.022+0000 7fef78ff9640 10 client.?.objecter ms_handle_connect 0x7fef7c0a98f0 2024-03-25T17:39:56.022 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.022+0000 7fef78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:56.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.028+0000 7fef78ff9640 10 client.?.objecter ms_handle_connect 0x7fef7c1430c0 2024-03-25T17:39:56.028 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.028+0000 7fef78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:56.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.039+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453791488 0 0) 0x7fef7c06e680 con 0x7fef7c0a98f0 2024-03-25T17:39:56.039 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.039+0000 7fef78ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:56.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.039+0000 7fef78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:56.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.039+0000 7fef78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:56.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.039+0000 7fef78ff9640 10 monclient(hunting): my global_id is 34664 2024-03-25T17:39:56.040 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.039+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef68004eb0 con 0x7fef7c0a98f0 2024-03-25T17:39:56.041 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.041+0000 7fef78ff9640 10 client.?.objecter ms_handle_connect 0x7fef7c13f710 2024-03-25T17:39:56.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.041+0000 7fef78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:39:56.110 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:01.126 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.117 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:01.134 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.123+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912533509 0 0) 0x7fef7c070210 con 0x7fef7c13f710 2024-03-25T17:39:56.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.123+0000 7fef78ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:56.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.123+0000 7fef78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:56.123 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.123+0000 7fef78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:56.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.123+0000 7fef78ff9640 10 monclient(hunting): my global_id is 4683 2024-03-25T17:39:56.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.123+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef7c06e680 con 0x7fef7c13f710 2024-03-25T17:39:56.155 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:01.172 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.171+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334336425 0 0) 0x7fef7c06dea0 con 0x7fef7c1430c0 2024-03-25T17:39:56.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.171+0000 7fef78ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:39:56.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.171+0000 7fef78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:39:56.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.171+0000 7fef78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:39:56.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.171+0000 7fef78ff9640 10 monclient(hunting): my global_id is 4711 2024-03-25T17:39:56.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.171+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef7c070210 con 0x7fef7c1430c0 2024-03-25T17:39:56.177 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:01.206 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.178+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3546147287 0 0) 0x7fef7c06e680 con 0x7fef7c13f710 2024-03-25T17:39:56.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.178+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef7c06dea0 con 0x7fef7c13f710 2024-03-25T17:39:56.178 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.178+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fef70001b00 con 0x7fef7c13f710 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.178+0000 7fef78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.178+0000 7fef78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.178+0000 7fef78ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:56.179 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.178+0000 7fef78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:39:56.184 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:01.212 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.187 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:01.204 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.187+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797599546 0 0) 0x7fef68004eb0 con 0x7fef7c0a98f0 2024-03-25T17:39:56.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.187+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef7c06e680 con 0x7fef7c0a98f0 2024-03-25T17:39:56.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.187+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fef6c001a30 con 0x7fef7c0a98f0 2024-03-25T17:39:56.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.187+0000 7fef78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:56.188 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.187+0000 7fef78ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.187+0000 7fef78ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:56.189 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:56.194 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.194+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3096351855 0 0) 0x7fef7c070210 con 0x7fef7c1430c0 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.194+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef68004eb0 con 0x7fef7c1430c0 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.194+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fef74001b90 con 0x7fef7c1430c0 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.194+0000 7fef78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.194+0000 7fef78ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.194+0000 7fef78ff9640 10 monclient(hunting): dump: 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:56.195 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:56.196 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:56.196 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:56.199 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:01.215 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 948137295 0 0) 0x7fef7c06dea0 con 0x7fef7c13f710 2024-03-25T17:39:56.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 >> v1:172.21.15.134:6790/0 conn(0x7fef7c0a98f0 legacy=0x7fef7c1dbf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:56.214 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 >> v1:172.21.15.159:6789/0 conn(0x7fef7c1430c0 legacy=0x7fef7c1e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 1 monclient: found mon.a 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7c1dc620 con 0x7fef7c13f710 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 10 monclient: _finish_auth 0 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 5 monclient: authenticate success, global_id 4683 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 10 monclient: _renew_subs 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef7c06f2e0 con 0x7fef7c13f710 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:39:56.215 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 10 monclient: _renew_subs 2024-03-25T17:39:56.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:56.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef7c139f00 con 0x7fef7c13f710 2024-03-25T17:39:56.216 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.213+0000 7fef822c5640 1 librados: init done 2024-03-25T17:39:56.216 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:01.243 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.214+0000 7fef822c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:56.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.214+0000 7fef822c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:56.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.214+0000 7fef822c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:56.217 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.214+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7c1db890 con 0x7fef7c13f710 2024-03-25T17:39:56.226 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:01.243 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:39:56.593 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.593+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef70001ca0 con 0x7fef7c13f710 2024-03-25T17:39:56.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.593+0000 7fef78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:39:56.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.593+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7fef70004a90 con 0x7fef7c13f710 2024-03-25T17:39:56.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.593+0000 7fef78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:39:56.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.593+0000 7fef78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:39:56.594 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.593+0000 7fef78ff9640 10 monclient: dump: 2024-03-25T17:39:56.594 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:39:56.594 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:39:56.595 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.595 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:39:56.595 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:39:56.595 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:39:56.595 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:39:56.595 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:39:56.596 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:39:56.596 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:39:56.596 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.593+0000 7fef7a7fc640 4 set_mon_vals no callback set 2024-03-25T17:39:56.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.594+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7fef7001aa60 con 0x7fef7c13f710 2024-03-25T17:39:56.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.594+0000 7fef7bfff640 10 monclient: get_auth_request con 0x7fef6806e0f0 auth_method 0 2024-03-25T17:39:56.597 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.595+0000 7fef78ff9640 10 client.4683.objecter ms_handle_connect 0x7fef6806e0f0 2024-03-25T17:39:56.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7fef70054640 con 0x7fef7c13f710 2024-03-25T17:39:56.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 10 client.4683.objecter ms_dispatch 0x7fef7c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:39:56.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 3 client.4683.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:39:56.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 3 client.4683.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:39:56.603 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T17:39:56.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef70050150 con 0x7fef7c13f710 2024-03-25T17:39:56.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:39:56.604 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.602+0000 7fef78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:39:56.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.850+0000 7fef822c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:56.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.850+0000 7fef822c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:56.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.850+0000 7fef822c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:39:56.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.850+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef7c06f470 con 0x7fef7c13f710 2024-03-25T17:39:56.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.902+0000 7fef78ff9640 1 -- 172.21.15.134:0/2576144731 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7fef70052080 con 0x7fef7c13f710 2024-03-25T17:39:56.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.902+0000 7fef78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:39:56.903 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.902+0000 7fef78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:39:56.903 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:39:56.903 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:39:56.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 10 librados: watch_flush enter 2024-03-25T17:39:56.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 10 librados: watch_flush exit 2024-03-25T17:39:56.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T17:39:56.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 10 client.4683.objecter successfully canceled tick 2024-03-25T17:39:56.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 >> v1:172.21.15.134:6800/610837766 conn(0x7fef6806e0f0 legacy=0x7fef680705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:56.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 10 monclient: shutdown 2024-03-25T17:39:56.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:39:56.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 >> v1:172.21.15.134:6789/0 conn(0x7fef7c13f710 legacy=0x7fef7c06ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:39:56.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 shutdown_connections 2024-03-25T17:39:56.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 >> 172.21.15.134:0/2576144731 conn(0x7fef7c0b7ea0 msgr2=0x7fef7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:39:56.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 shutdown_connections 2024-03-25T17:39:56.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 1 -- 172.21.15.134:0/2576144731 wait complete. 2024-03-25T17:39:56.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:39:56.905+0000 7fef822c5640 1 librados: shutdown 2024-03-25T17:39:56.918 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:03.724 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:03.823 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.823+0000 7ff76a567640 10 monclient: get_monmap_and_config 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.823+0000 7ff76a567640 10 monclient: build_initial_monmap 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.823+0000 7ff76a567640 10 monclient: monmap: 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:40:03.824115+0000 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:40:03.824115+0000 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:40:03.824 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:40:03.825 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.824+0000 7ff76a567640 1 Processor -- start 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.824+0000 7ff76a567640 1 -- start start 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.824+0000 7ff76a567640 10 monclient: init 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.824+0000 7ff76a567640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.824+0000 7ff76a567640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.824+0000 7ff76a567640 10 monclient(hunting): picked mon.noname-c con 0x7ff7640b1ef0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.825+0000 7ff76a567640 10 monclient(hunting): picked mon.noname-b con 0x7ff7641473a0 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.825+0000 7ff76a567640 10 monclient(hunting): picked mon.noname-a con 0x7ff764150070 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.825+0000 7ff76a567640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7640b2570 con 0x7ff764150070 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.825+0000 7ff76a567640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764146c20 con 0x7ff7640b1ef0 2024-03-25T17:40:03.826 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.825+0000 7ff76a567640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764146e80 con 0x7ff7641473a0 2024-03-25T17:40:03.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.825+0000 7ff76a567640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:03.827 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.825+0000 7ff76a567640 10 monclient(hunting): authenticate will time out at 5957.087906s 2024-03-25T17:40:03.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.838+0000 7ff7637fe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7ff7641473a0 0x7ff764143b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:46422/0 (socket says 172.21.15.134:46422) 2024-03-25T17:40:03.839 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.838+0000 7ff7637fe640 1 -- 172.21.15.134:0/1134721528 learned_addr learned my addr 172.21.15.134:0/1134721528 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:03.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.853+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1689070896 0 0) 0x7ff764146e80 con 0x7ff7641473a0 2024-03-25T17:40:03.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.853+0000 7ff762ffd640 10 monclient(hunting): _negotiate 2024-03-25T17:40:03.854 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.853+0000 7ff762ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:03.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.853+0000 7ff762ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:03.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.853+0000 7ff762ffd640 10 monclient(hunting): my global_id is 34667 2024-03-25T17:40:03.855 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.854+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74c003520 con 0x7ff7641473a0 2024-03-25T17:40:03.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1779882146 0 0) 0x7ff74c003520 con 0x7ff7641473a0 2024-03-25T17:40:03.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 >> v1:172.21.15.159:6789/0 conn(0x7ff7640b1ef0 legacy=0x7ff7640b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:03.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 >> v1:172.21.15.134:6789/0 conn(0x7ff764150070 legacy=0x7ff764152460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:03.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 10 monclient: _finish_hunting 0 2024-03-25T17:40:03.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 1 monclient: found mon.noname-b 2024-03-25T17:40:03.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:03.861 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76406e1b0 con 0x7ff7641473a0 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 10 monclient: _finish_auth 0 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 10 monclient: _check_auth_tickets 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff758001690 con 0x7ff7641473a0 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff76a567640 5 monclient: authenticate success, global_id 34667 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff76a567640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff762ffd640 10 monclient: dump: 2024-03-25T17:40:03.862 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:03.863 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.860+0000 7ff76a567640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:03.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7580017f0 con 0x7ff7641473a0 2024-03-25T17:40:03.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff762ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:03.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff762ffd640 1 -- 172.21.15.134:0/1134721528 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff758001c00 con 0x7ff7641473a0 2024-03-25T17:40:03.879 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff762ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:03.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff762ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:03.881 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff762ffd640 10 monclient: dump: 2024-03-25T17:40:03.881 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:03.881 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:03.881 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.881 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.881 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:03.882 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:03.882 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:03.882 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:03.882 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:03.882 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:03.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff76a567640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:03.882 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.878+0000 7ff76a567640 4 set_mon_vals no callback set 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.879+0000 7ff76a567640 10 monclient: shutdown 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.879+0000 7ff76a567640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.879+0000 7ff76a567640 1 -- 172.21.15.134:0/1134721528 >> v1:172.21.15.134:6790/0 conn(0x7ff7641473a0 legacy=0x7ff764143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.879+0000 7ff76a567640 1 -- 172.21.15.134:0/1134721528 shutdown_connections 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.879+0000 7ff76a567640 1 -- 172.21.15.134:0/1134721528 >> 172.21.15.134:0/1134721528 conn(0x7ff7640b7ea0 msgr2=0x7ff7640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.879+0000 7ff76a567640 1 -- 172.21.15.134:0/1134721528 shutdown_connections 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.879+0000 7ff76a567640 1 -- 172.21.15.134:0/1134721528 wait complete. 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.880+0000 7ff76a567640 10 monclient: build_initial_monmap 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.880+0000 7ff76a567640 10 monclient: monmap: 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:40:03.885 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 1 librados: starting msgr at 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 1 librados: starting objecter 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 1 Processor -- start 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 1 -- start start 2024-03-25T17:40:03.886 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 1 librados: setting wanted keys 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 1 librados: calling monclient init 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 10 monclient: init 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 10 monclient(hunting): picked mon.noname-c con 0x7ff7640b1ef0 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 10 monclient(hunting): picked mon.noname-a con 0x7ff7641473a0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.881+0000 7ff76a567640 10 monclient(hunting): picked mon.noname-b con 0x7ff764150070 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.882+0000 7ff76a567640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76406f810 con 0x7ff7641473a0 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.882+0000 7ff76a567640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764070690 con 0x7ff764150070 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.882+0000 7ff76a567640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff764144700 con 0x7ff7640b1ef0 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.882+0000 7ff76a567640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.882+0000 7ff76a567640 10 monclient(hunting): authenticate will time out at 5957.144801s 2024-03-25T17:40:03.887 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.883+0000 7ff768b04640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7ff764150070 0x7ff7641e7720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:41540/0 (socket says 172.21.15.134:41540) 2024-03-25T17:40:03.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.883+0000 7ff768b04640 1 -- 172.21.15.134:0/3171496555 learned_addr learned my addr 172.21.15.134:0/3171496555 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:03.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.883+0000 7ff760ff9640 10 client.?.objecter ms_handle_connect 0x7ff764150070 2024-03-25T17:40:03.888 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.883+0000 7ff760ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:03.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.895+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308466751 0 0) 0x7ff764070690 con 0x7ff764150070 2024-03-25T17:40:03.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.895+0000 7ff760ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:40:03.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.895+0000 7ff760ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:03.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.895+0000 7ff760ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:03.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.895+0000 7ff760ff9640 10 monclient(hunting): my global_id is 4717 2024-03-25T17:40:03.896 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.895+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff750004eb0 con 0x7ff764150070 2024-03-25T17:40:03.898 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.898+0000 7ff760ff9640 10 client.?.objecter ms_handle_connect 0x7ff7640b1ef0 2024-03-25T17:40:03.898 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.898+0000 7ff760ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:03.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.906+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961613561 0 0) 0x7ff764144700 con 0x7ff7640b1ef0 2024-03-25T17:40:03.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.906+0000 7ff760ff9640 10 monclient(hunting): _negotiate 2024-03-25T17:40:03.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.906+0000 7ff760ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:03.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.906+0000 7ff760ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:03.906 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.906+0000 7ff760ff9640 10 monclient(hunting): my global_id is 34670 2024-03-25T17:40:03.907 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.906+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff764070690 con 0x7ff7640b1ef0 2024-03-25T17:40:03.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 10 client.?.objecter ms_handle_connect 0x7ff7641473a0 2024-03-25T17:40:03.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:03.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2589248875 0 0) 0x7ff764070690 con 0x7ff7640b1ef0 2024-03-25T17:40:03.917 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff764144700 con 0x7ff7640b1ef0 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff754001930 con 0x7ff7640b1ef0 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 10 monclient(hunting): dump: 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:03.918 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.916+0000 7ff760ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.917+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743326885 0 0) 0x7ff750004eb0 con 0x7ff764150070 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.917+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff764070690 con 0x7ff764150070 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.917+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff75c001bd0 con 0x7ff764150070 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.917+0000 7ff760ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.917+0000 7ff760ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.917+0000 7ff760ff9640 10 monclient(hunting): dump: 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:03.919 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:03.920 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:03.920 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:03.920 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1937378480 0 0) 0x7ff764144700 con 0x7ff7640b1ef0 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 >> v1:172.21.15.159:6789/0 conn(0x7ff764150070 legacy=0x7ff7641e7720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 >> v1:172.21.15.134:6789/0 conn(0x7ff7641473a0 legacy=0x7ff764137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 10 monclient: _finish_hunting 0 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 1 monclient: found mon.c 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76406fa60 con 0x7ff7640b1ef0 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 10 monclient: _finish_auth 0 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 10 monclient: _check_auth_tickets 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff760ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 5 monclient: authenticate success, global_id 34670 2024-03-25T17:40:03.929 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 10 monclient: _renew_subs 2024-03-25T17:40:03.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:03.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7640708d0 con 0x7ff7640b1ef0 2024-03-25T17:40:03.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 10 client.34670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:03.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 10 monclient: _renew_subs 2024-03-25T17:40:03.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:03.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76413b020 con 0x7ff7640b1ef0 2024-03-25T17:40:03.930 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.928+0000 7ff76a567640 1 librados: init done 2024-03-25T17:40:03.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.929+0000 7ff76a567640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:03.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.929+0000 7ff76a567640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:03.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.929+0000 7ff76a567640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:03.931 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:03.929+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7640627b0 con 0x7ff7640b1ef0 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.241+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff754003610 con 0x7ff7640b1ef0 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.241+0000 7ff760ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.241+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7ff754003960 con 0x7ff7640b1ef0 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.241+0000 7ff760ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.241+0000 7ff760ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.241+0000 7ff760ff9640 10 monclient: dump: 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:04.242 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:04.243 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:04.243 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:04.243 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:04.243 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:04.243 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:04.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.241+0000 7ff7627fc640 4 set_mon_vals no callback set 2024-03-25T17:40:04.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.242+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7ff754003ac0 con 0x7ff7640b1ef0 2024-03-25T17:40:04.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.243+0000 7ff7637fe640 10 monclient: get_auth_request con 0x7ff75006ded0 auth_method 0 2024-03-25T17:40:04.244 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.243+0000 7ff760ff9640 10 client.34670.objecter ms_handle_connect 0x7ff75006ded0 2024-03-25T17:40:04.249 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7ff754053680 con 0x7ff7640b1ef0 2024-03-25T17:40:04.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 10 client.34670.objecter ms_dispatch 0x7ff764000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:40:04.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 3 client.34670.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:40:04.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 3 client.34670.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:40:04.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 20 client.34670.objecter dump_active .. 0 homeless 2024-03-25T17:40:04.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff75404f190 con 0x7ff7640b1ef0 2024-03-25T17:40:04.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:04.250 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.249+0000 7ff760ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:04.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.493+0000 7ff76a567640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:04.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.493+0000 7ff76a567640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:04.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.493+0000 7ff76a567640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:04.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.493+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7641435a0 con 0x7ff7640b1ef0 2024-03-25T17:40:04.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.515+0000 7ff760ff9640 1 -- 172.21.15.134:0/3171496555 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7ff7540510c0 con 0x7ff7640b1ef0 2024-03-25T17:40:04.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.515+0000 7ff760ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:04.516 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.515+0000 7ff760ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:04.517 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:40:04.517 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:04.518 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 10 librados: watch_flush enter 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 10 librados: watch_flush exit 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 20 client.34670.objecter shutdown clearing up homeless session... 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 10 client.34670.objecter successfully canceled tick 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 >> v1:172.21.15.134:6800/610837766 conn(0x7ff75006ded0 legacy=0x7ff750070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 10 monclient: shutdown 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.517+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 >> v1:172.21.15.134:6790/0 conn(0x7ff7640b1ef0 legacy=0x7ff76406ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.518+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 shutdown_connections 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.518+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 >> 172.21.15.134:0/3171496555 conn(0x7ff7640b7ea0 msgr2=0x7ff764142e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.518+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 shutdown_connections 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.518+0000 7ff76a567640 1 -- 172.21.15.134:0/3171496555 wait complete. 2024-03-25T17:40:04.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:04.518+0000 7ff76a567640 1 librados: shutdown 2024-03-25T17:40:04.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:11.337 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:11.440 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.440+0000 7f4f3d177640 10 monclient: get_monmap_and_config 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.440+0000 7f4f3d177640 10 monclient: build_initial_monmap 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.440+0000 7f4f3d177640 10 monclient: monmap: 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:40:11.441293+0000 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:40:11.441293+0000 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:40:11.441 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.441+0000 7f4f3d177640 1 Processor -- start 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.441+0000 7f4f3d177640 1 -- start start 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.441+0000 7f4f3d177640 10 monclient: init 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.441+0000 7f4f3d177640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.441+0000 7f4f3d177640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 10 monclient(hunting): picked mon.noname-b con 0x7f4f380ae6e0 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 10 monclient(hunting): picked mon.noname-a con 0x7f4f3806f080 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 10 monclient(hunting): picked mon.noname-c con 0x7f4f380704e0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f380718d0 con 0x7f4f3806f080 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3806fa30 con 0x7f4f380704e0 2024-03-25T17:40:11.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38070090 con 0x7f4f380ae6e0 2024-03-25T17:40:11.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:11.444 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.442+0000 7f4f3d177640 10 monclient(hunting): authenticate will time out at 5964.705099s 2024-03-25T17:40:11.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.445+0000 7f4f3759e640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f4f380704e0 0x7f4f3814a910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:37720/0 (socket says 172.21.15.134:37720) 2024-03-25T17:40:11.446 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.445+0000 7f4f3759e640 1 -- 172.21.15.134:0/3245964324 learned_addr learned my addr 172.21.15.134:0/3245964324 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.453+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365764731 0 0) 0x7f4f380718d0 con 0x7f4f3806f080 2024-03-25T17:40:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.453+0000 7f4f35d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:40:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.453+0000 7f4f35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.453+0000 7f4f35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.453+0000 7f4f35d9b640 10 monclient(hunting): my global_id is 4692 2024-03-25T17:40:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.453+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f20003520 con 0x7f4f3806f080 2024-03-25T17:40:11.454 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.454+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252045140 0 0) 0x7f4f3806fa30 con 0x7f4f380704e0 2024-03-25T17:40:11.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.454+0000 7f4f35d9b640 10 monclient(hunting): _negotiate 2024-03-25T17:40:11.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.454+0000 7f4f35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:11.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.454+0000 7f4f35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:11.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.454+0000 7f4f35d9b640 10 monclient(hunting): my global_id is 4720 2024-03-25T17:40:11.455 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.454+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f380718d0 con 0x7f4f380704e0 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150389408 0 0) 0x7f4f20003520 con 0x7f4f3806f080 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 >> v1:172.21.15.134:6790/0 conn(0x7f4f380ae6e0 legacy=0x7f4f3813c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 >> v1:172.21.15.159:6789/0 conn(0x7f4f380704e0 legacy=0x7f4f3814a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 10 monclient: _finish_hunting 0 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 1 monclient: found mon.noname-a 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:40:11.461 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3806f510 con 0x7f4f3806f080 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 10 monclient: _finish_auth 0 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 10 monclient: _check_auth_tickets 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4f2c001690 con 0x7f4f3806f080 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f35d9b640 10 monclient: dump: 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:11.462 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:11.463 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:11.463 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:11.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f3d177640 5 monclient: authenticate success, global_id 4692 2024-03-25T17:40:11.463 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.460+0000 7f4f3d177640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:11.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f2c0017f0 con 0x7f4f3806f080 2024-03-25T17:40:11.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f35d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:11.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f35d9b640 1 -- 172.21.15.134:0/3245964324 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4f2c001c00 con 0x7f4f3806f080 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f35d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f35d9b640 10 monclient: dump: 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f3d177640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:11.469 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.468+0000 7f4f3d177640 4 set_mon_vals no callback set 2024-03-25T17:40:11.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.469+0000 7f4f3d177640 10 monclient: shutdown 2024-03-25T17:40:11.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.469+0000 7f4f3d177640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:11.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.469+0000 7f4f3d177640 1 -- 172.21.15.134:0/3245964324 >> v1:172.21.15.134:6789/0 conn(0x7f4f3806f080 legacy=0x7f4f38071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:11.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.469+0000 7f4f3d177640 1 -- 172.21.15.134:0/3245964324 shutdown_connections 2024-03-25T17:40:11.470 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.469+0000 7f4f3d177640 1 -- 172.21.15.134:0/3245964324 >> 172.21.15.134:0/3245964324 conn(0x7f4f3806b630 msgr2=0x7f4f380b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:11.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.469+0000 7f4f3d177640 1 -- 172.21.15.134:0/3245964324 shutdown_connections 2024-03-25T17:40:11.471 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.469+0000 7f4f3d177640 1 -- 172.21.15.134:0/3245964324 wait complete. 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.470+0000 7f4f3d177640 10 monclient: build_initial_monmap 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.470+0000 7f4f3d177640 10 monclient: monmap: 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:11.472 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.470+0000 7f4f3d177640 1 librados: starting msgr at 2024-03-25T17:40:11.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.470+0000 7f4f3d177640 1 librados: starting objecter 2024-03-25T17:40:11.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.471+0000 7f4f3d177640 1 Processor -- start 2024-03-25T17:40:11.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.471+0000 7f4f3d177640 1 -- start start 2024-03-25T17:40:11.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.471+0000 7f4f3d177640 1 librados: setting wanted keys 2024-03-25T17:40:11.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.471+0000 7f4f3d177640 1 librados: calling monclient init 2024-03-25T17:40:11.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.471+0000 7f4f3d177640 10 monclient: init 2024-03-25T17:40:11.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 10 monclient(hunting): picked mon.noname-a con 0x7f4f380704e0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 10 monclient(hunting): picked mon.noname-c con 0x7f4f380ae6e0 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 10 monclient(hunting): picked mon.noname-b con 0x7f4f380aaf50 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f380aa7c0 con 0x7f4f380704e0 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f381df710 con 0x7f4f380aaf50 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38142010 con 0x7f4f380ae6e0 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:11.474 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.472+0000 7f4f3d177640 10 monclient(hunting): authenticate will time out at 5964.735133s 2024-03-25T17:40:11.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.479+0000 7f4f36d9d640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f4f380704e0 0x7f4f381df000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:44620/0 (socket says 172.21.15.134:44620) 2024-03-25T17:40:11.479 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.479+0000 7f4f36d9d640 1 -- 172.21.15.134:0/1084200989 learned_addr learned my addr 172.21.15.134:0/1084200989 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:11.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.481+0000 7f4f1f7fe640 10 client.?.objecter ms_handle_connect 0x7f4f380704e0 2024-03-25T17:40:11.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.481+0000 7f4f1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:11.482 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.482+0000 7f4f1f7fe640 10 client.?.objecter ms_handle_connect 0x7f4f380aaf50 2024-03-25T17:40:11.483 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.482+0000 7f4f1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:11.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.493+0000 7f4f1f7fe640 10 client.?.objecter ms_handle_connect 0x7f4f380ae6e0 2024-03-25T17:40:11.493 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.493+0000 7f4f1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:11.494 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.494+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3663964878 0 0) 0x7f4f380aa7c0 con 0x7f4f380704e0 2024-03-25T17:40:11.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.494+0000 7f4f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:11.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.494+0000 7f4f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:11.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.494+0000 7f4f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:11.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.494+0000 7f4f1f7fe640 10 monclient(hunting): my global_id is 4695 2024-03-25T17:40:11.495 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.494+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f24004eb0 con 0x7f4f380704e0 2024-03-25T17:40:11.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.499+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285321505 0 0) 0x7f4f381df710 con 0x7f4f380aaf50 2024-03-25T17:40:11.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.499+0000 7f4f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:11.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.499+0000 7f4f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:11.499 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.499+0000 7f4f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:11.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.499+0000 7f4f1f7fe640 10 monclient(hunting): my global_id is 4723 2024-03-25T17:40:11.500 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.499+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f380aa7c0 con 0x7f4f380aaf50 2024-03-25T17:40:11.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.502+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912971339 0 0) 0x7f4f38142010 con 0x7f4f380ae6e0 2024-03-25T17:40:11.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.502+0000 7f4f1f7fe640 10 monclient(hunting): _negotiate 2024-03-25T17:40:11.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.502+0000 7f4f1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:11.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.502+0000 7f4f1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:11.502 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.502+0000 7f4f1f7fe640 10 monclient(hunting): my global_id is 34676 2024-03-25T17:40:11.503 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.502+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f381df710 con 0x7f4f380ae6e0 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.510+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 859038770 0 0) 0x7f4f24004eb0 con 0x7f4f380704e0 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.511+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f38142010 con 0x7f4f380704e0 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.511+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4f28001990 con 0x7f4f380704e0 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.511+0000 7f4f1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.511+0000 7f4f1f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.511+0000 7f4f1f7fe640 10 monclient(hunting): dump: 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:11.511 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:11.512 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.511+0000 7f4f1f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:11.514 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.514+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143619223 0 0) 0x7f4f380aa7c0 con 0x7f4f380aaf50 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.514+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f24004eb0 con 0x7f4f380aaf50 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.514+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4f30001c70 con 0x7f4f380aaf50 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.514+0000 7f4f1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.514+0000 7f4f1f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.514+0000 7f4f1f7fe640 10 monclient(hunting): dump: 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:11.515 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:11.516 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:11.516 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:11.516 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4220771874 0 0) 0x7f4f38142010 con 0x7f4f380704e0 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 >> v1:172.21.15.134:6790/0 conn(0x7f4f380ae6e0 legacy=0x7f4f381386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 >> v1:172.21.15.159:6789/0 conn(0x7f4f380aaf50 legacy=0x7f4f381e7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 10 monclient: _finish_hunting 0 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 1 monclient: found mon.a 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f381421a0 con 0x7f4f380704e0 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 10 monclient: _finish_auth 0 2024-03-25T17:40:11.519 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 10 monclient: _check_auth_tickets 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 5 monclient: authenticate success, global_id 4695 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 10 monclient: _renew_subs 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f38138dc0 con 0x7f4f380704e0 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 10 monclient: _renew_subs 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f380ab970 con 0x7f4f380704e0 2024-03-25T17:40:11.520 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.518+0000 7f4f3d177640 1 librados: init done 2024-03-25T17:40:11.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.519+0000 7f4f3d177640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:11.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.519+0000 7f4f3d177640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:11.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.519+0000 7f4f3d177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:11.521 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.519+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f381423b0 con 0x7f4f380704e0 2024-03-25T17:40:11.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f28003610 con 0x7f4f380704e0 2024-03-25T17:40:11.845 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4f280039e0 con 0x7f4f380704e0 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f1f7fe640 10 monclient: dump: 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:11.846 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:11.847 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:11.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f3559a640 4 set_mon_vals no callback set 2024-03-25T17:40:11.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.845+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f4f28003c00 con 0x7f4f380704e0 2024-03-25T17:40:11.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.846+0000 7f4f3659c640 10 monclient: get_auth_request con 0x7f4f2406df70 auth_method 0 2024-03-25T17:40:11.847 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.846+0000 7f4f1f7fe640 10 client.4695.objecter ms_handle_connect 0x7f4f2406df70 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f4f28054850 con 0x7f4f380704e0 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 10 client.4695.objecter ms_dispatch 0x7f4f38000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 3 client.4695.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 3 client.4695.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f28050360 con 0x7f4f380704e0 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:11.851 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:11.850+0000 7f4f1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:12.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.101+0000 7f4f3d177640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:12.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.101+0000 7f4f3d177640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:12.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.101+0000 7f4f3d177640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:12.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.101+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f380ab5d0 con 0x7f4f380704e0 2024-03-25T17:40:12.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.131+0000 7f4f1f7fe640 1 -- 172.21.15.134:0/1084200989 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f4f28052290 con 0x7f4f380704e0 2024-03-25T17:40:12.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.131+0000 7f4f1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:12.132 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.131+0000 7f4f1f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:12.132 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:40:12.132 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:12.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 10 librados: watch_flush enter 2024-03-25T17:40:12.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 10 librados: watch_flush exit 2024-03-25T17:40:12.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 10 client.4695.objecter successfully canceled tick 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 >> v1:172.21.15.134:6800/610837766 conn(0x7f4f2406df70 legacy=0x7f4f24070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 10 monclient: shutdown 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 >> v1:172.21.15.134:6789/0 conn(0x7f4f380704e0 legacy=0x7f4f381df000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 shutdown_connections 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.133+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 >> 172.21.15.134:0/1084200989 conn(0x7f4f3806b630 msgr2=0x7f4f3813c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.134+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 shutdown_connections 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.134+0000 7f4f3d177640 1 -- 172.21.15.134:0/1084200989 wait complete. 2024-03-25T17:40:12.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:12.134+0000 7f4f3d177640 1 librados: shutdown 2024-03-25T17:40:12.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:16.163 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:21.179 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.171 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:21.187 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.199 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:21.215 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.210 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:21.239 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.231 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:21.247 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.232 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:21.261 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.244 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:21.260 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.259 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:21.288 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:16.277 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:21.294 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:18.955 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:19.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.070+0000 7f22c8055640 10 monclient: get_monmap_and_config 2024-03-25T17:40:19.071 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.070+0000 7f22c8055640 10 monclient: build_initial_monmap 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.070+0000 7f22c8055640 10 monclient: monmap: 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:40:19.071511+0000 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:40:19.071511+0000 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:19.072 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.071+0000 7f22c8055640 1 Processor -- start 2024-03-25T17:40:19.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.071+0000 7f22c8055640 1 -- start start 2024-03-25T17:40:19.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.071+0000 7f22c8055640 10 monclient: init 2024-03-25T17:40:19.073 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 10 monclient(hunting): picked mon.noname-b con 0x7f22c00a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 10 monclient(hunting): picked mon.noname-c con 0x7f22c013f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 10 monclient(hunting): picked mon.noname-a con 0x7f22c01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c0142520 con 0x7f22c01430c0 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c00b28a0 con 0x7f22c013f710 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c00b2a30 con 0x7f22c00a98f0 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:19.074 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.072+0000 7f22c8055640 10 monclient(hunting): authenticate will time out at 5972.335131s 2024-03-25T17:40:19.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.080+0000 7f22c65f2640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f22c01430c0 0x7f22c014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49118/0 (socket says 172.21.15.134:49118) 2024-03-25T17:40:19.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.080+0000 7f22c65f2640 1 -- 172.21.15.134:0/3318426415 learned_addr learned my addr 172.21.15.134:0/3318426415 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:19.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.080+0000 7f22c55f0640 1 --1- 172.21.15.134:0/3318426415 >> v1:172.21.15.159:6789/0 conn(0x7f22c013f710 0x7f22c01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:42498/0 (socket says 172.21.15.134:42498) 2024-03-25T17:40:19.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.091+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460136022 0 0) 0x7f22c0142520 con 0x7f22c01430c0 2024-03-25T17:40:19.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.091+0000 7f22c4def640 10 monclient(hunting): _negotiate 2024-03-25T17:40:19.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.091+0000 7f22c4def640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:19.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.091+0000 7f22c4def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:19.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.091+0000 7f22c4def640 10 monclient(hunting): my global_id is 4698 2024-03-25T17:40:19.092 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.091+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ac003520 con 0x7f22c01430c0 2024-03-25T17:40:19.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.093+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 833477423 0 0) 0x7f22c00b2a30 con 0x7f22c00a98f0 2024-03-25T17:40:19.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.093+0000 7f22c4def640 10 monclient(hunting): _negotiate 2024-03-25T17:40:19.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.093+0000 7f22c4def640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:19.094 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.093+0000 7f22c4def640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:19.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.093+0000 7f22c4def640 10 monclient(hunting): my global_id is 34679 2024-03-25T17:40:19.095 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.093+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c0142520 con 0x7f22c00a98f0 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975174502 0 0) 0x7f22ac003520 con 0x7f22c01430c0 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 >> v1:172.21.15.134:6790/0 conn(0x7f22c00a98f0 legacy=0x7f22c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 >> v1:172.21.15.159:6789/0 conn(0x7f22c013f710 legacy=0x7f22c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 10 monclient: _finish_hunting 0 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 1 monclient: found mon.noname-a 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c00b44c0 con 0x7f22c01430c0 2024-03-25T17:40:19.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 10 monclient: _finish_auth 0 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 10 monclient: _check_auth_tickets 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c8055640 5 monclient: authenticate success, global_id 4698 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c8055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f22b80019d0 con 0x7f22c01430c0 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c4def640 10 monclient: dump: 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:19.099 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:19.100 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.097+0000 7f22c8055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22b8001c20 con 0x7f22c01430c0 2024-03-25T17:40:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c4def640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c4def640 1 -- 172.21.15.134:0/3318426415 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f22b8001fb0 con 0x7f22c01430c0 2024-03-25T17:40:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c8055640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:19.117 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c8055640 4 set_mon_vals no callback set 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c4def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c4def640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.116+0000 7f22c4def640 10 monclient: dump: 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:19.118 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:19.119 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:19.119 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:19.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.117+0000 7f22c8055640 10 monclient: shutdown 2024-03-25T17:40:19.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.117+0000 7f22c8055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:19.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.117+0000 7f22c8055640 1 -- 172.21.15.134:0/3318426415 >> v1:172.21.15.134:6789/0 conn(0x7f22c01430c0 legacy=0x7f22c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:19.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.117+0000 7f22c8055640 1 -- 172.21.15.134:0/3318426415 shutdown_connections 2024-03-25T17:40:19.120 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.117+0000 7f22c8055640 1 -- 172.21.15.134:0/3318426415 >> 172.21.15.134:0/3318426415 conn(0x7f22c00b7ea0 msgr2=0x7f22c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.118+0000 7f22c8055640 1 -- 172.21.15.134:0/3318426415 shutdown_connections 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.118+0000 7f22c8055640 1 -- 172.21.15.134:0/3318426415 wait complete. 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 10 monclient: build_initial_monmap 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 10 monclient: monmap: 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:19.121 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 1 librados: starting msgr at 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 1 librados: starting objecter 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 1 Processor -- start 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 1 -- start start 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 1 librados: setting wanted keys 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 1 librados: calling monclient init 2024-03-25T17:40:19.122 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.119+0000 7f22c8055640 10 monclient: init 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 10 monclient(hunting): picked mon.noname-b con 0x7f22c00a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 10 monclient(hunting): picked mon.noname-c con 0x7f22c013f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 10 monclient(hunting): picked mon.noname-a con 0x7f22c01430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c0133d30 con 0x7f22c01430c0 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c01397f0 con 0x7f22c00a98f0 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c0136ae0 con 0x7f22c013f710 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:19.124 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.120+0000 7f22c8055640 10 monclient(hunting): authenticate will time out at 5972.383740s 2024-03-25T17:40:19.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.126+0000 7f22c5df1640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f22c00a98f0 0x7f22c01db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:42538/0 (socket says 172.21.15.134:42538) 2024-03-25T17:40:19.126 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.126+0000 7f22c5df1640 1 -- 172.21.15.134:0/423799012 learned_addr learned my addr 172.21.15.134:0/423799012 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:19.128 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.128+0000 7f22b67fc640 10 client.?.objecter ms_handle_connect 0x7f22c00a98f0 2024-03-25T17:40:19.129 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.128+0000 7f22b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:19.130 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.130+0000 7f22b67fc640 10 client.?.objecter ms_handle_connect 0x7f22c013f710 2024-03-25T17:40:19.131 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.130+0000 7f22b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:19.134 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.134+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828987770 0 0) 0x7f22c01397f0 con 0x7f22c00a98f0 2024-03-25T17:40:19.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.134+0000 7f22b67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:19.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.134+0000 7f22b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:19.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.134+0000 7f22b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:19.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.134+0000 7f22b67fc640 10 monclient(hunting): my global_id is 4729 2024-03-25T17:40:19.135 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.134+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a4004eb0 con 0x7f22c00a98f0 2024-03-25T17:40:19.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.139+0000 7f22b67fc640 10 client.?.objecter ms_handle_connect 0x7f22c01430c0 2024-03-25T17:40:19.140 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.139+0000 7f22b67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:19.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.147+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597975593 0 0) 0x7f22c0136ae0 con 0x7f22c013f710 2024-03-25T17:40:19.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.147+0000 7f22b67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:19.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.147+0000 7f22b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:19.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.147+0000 7f22b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:19.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.147+0000 7f22b67fc640 10 monclient(hunting): my global_id is 34682 2024-03-25T17:40:19.148 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.147+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c01397f0 con 0x7f22c013f710 2024-03-25T17:40:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.156+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580193367 0 0) 0x7f22c0133d30 con 0x7f22c01430c0 2024-03-25T17:40:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.156+0000 7f22b67fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.156+0000 7f22b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.156+0000 7f22b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.156+0000 7f22b67fc640 10 monclient(hunting): my global_id is 4701 2024-03-25T17:40:19.157 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.156+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22c0136ae0 con 0x7f22c01430c0 2024-03-25T17:40:19.159 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.159+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1787343277 0 0) 0x7f22c01397f0 con 0x7f22c013f710 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.159+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22c0133d30 con 0x7f22c013f710 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.159+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f22b00018a0 con 0x7f22c013f710 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.159+0000 7f22b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.159+0000 7f22b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.159+0000 7f22b67fc640 10 monclient(hunting): dump: 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:19.160 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:19.161 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:19.161 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:19.161 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:19.161 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:19.161 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.159+0000 7f22b67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:19.164 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.164+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678200504 0 0) 0x7f22a4004eb0 con 0x7f22c00a98f0 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.164+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22c01397f0 con 0x7f22c00a98f0 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.164+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f22bc001a10 con 0x7f22c00a98f0 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.164+0000 7f22b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.164+0000 7f22b67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.164+0000 7f22b67fc640 10 monclient(hunting): dump: 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.165 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:19.166 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:19.166 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:19.166 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:19.166 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:19.166 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 320220941 0 0) 0x7f22c0133d30 con 0x7f22c013f710 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 >> v1:172.21.15.159:6789/0 conn(0x7f22c00a98f0 legacy=0x7f22c01db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 >> v1:172.21.15.134:6789/0 conn(0x7f22c01430c0 legacy=0x7f22c01db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 1 monclient: found mon.c 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c01dc070 con 0x7f22c013f710 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:19.171 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 5 monclient: authenticate success, global_id 34682 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 10 monclient: _renew_subs 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c0139160 con 0x7f22c013f710 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 10 client.34682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 10 monclient: _renew_subs 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c0137c10 con 0x7f22c013f710 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.169+0000 7f22c8055640 1 librados: init done 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.170+0000 7f22c8055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:19.172 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.170+0000 7f22c8055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:19.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.170+0000 7f22c8055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:19.173 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.170+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c00627b0 con 0x7f22c013f710 2024-03-25T17:40:19.465 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.465+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22b0001d50 con 0x7f22c013f710 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.465+0000 7f22b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.465+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f22b00049c0 con 0x7f22c013f710 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.465+0000 7f22b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.465+0000 7f22b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.465+0000 7f22b67fc640 10 monclient: dump: 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:19.466 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:19.467 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.465+0000 7f22b7fff640 4 set_mon_vals no callback set 2024-03-25T17:40:19.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.466+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f22b0004be0 con 0x7f22c013f710 2024-03-25T17:40:19.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.466+0000 7f22c5df1640 10 monclient: get_auth_request con 0x7f22a406df20 auth_method 0 2024-03-25T17:40:19.468 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.466+0000 7f22b67fc640 10 client.34682.objecter ms_handle_connect 0x7f22a406df20 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.471+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f22b0054700 con 0x7f22c013f710 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.471+0000 7f22b67fc640 10 client.34682.objecter ms_dispatch 0x7f22c0000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.471+0000 7f22b67fc640 3 client.34682.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.471+0000 7f22b67fc640 3 client.34682.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.471+0000 7f22b67fc640 20 client.34682.objecter dump_active .. 0 homeless 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.472+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22b0050040 con 0x7f22c013f710 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.472+0000 7f22b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:19.473 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.472+0000 7f22b67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:19.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.698+0000 7f22c8055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:19.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.698+0000 7f22c8055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:19.699 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.698+0000 7f22c8055640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:19.700 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.698+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22c013cf80 con 0x7f22c013f710 2024-03-25T17:40:19.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.731+0000 7f22b67fc640 1 -- 172.21.15.134:0/423799012 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f22b0052140 con 0x7f22c013f710 2024-03-25T17:40:19.731 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.731+0000 7f22b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:19.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.731+0000 7f22b67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:19.732 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:40:19.732 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:19.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.733+0000 7f22c8055640 10 librados: watch_flush enter 2024-03-25T17:40:19.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 10 librados: watch_flush exit 2024-03-25T17:40:19.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 20 client.34682.objecter shutdown clearing up homeless session... 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 10 client.34682.objecter successfully canceled tick 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 >> v1:172.21.15.134:6800/610837766 conn(0x7f22a406df20 legacy=0x7f22a40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 10 monclient: shutdown 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 >> v1:172.21.15.134:6790/0 conn(0x7f22c013f710 legacy=0x7f22c01363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 shutdown_connections 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 >> 172.21.15.134:0/423799012 conn(0x7f22c00b7ea0 msgr2=0x7f22c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 shutdown_connections 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 1 -- 172.21.15.134:0/423799012 wait complete. 2024-03-25T17:40:19.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:19.734+0000 7f22c8055640 1 librados: shutdown 2024-03-25T17:40:19.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:26.056 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:31.072 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.062 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:31.078 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.069 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:31.098 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.072 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:31.101 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.081 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:31.098 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.082 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:31.099 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.086 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:31.102 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.092 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:31.120 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.104 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:31.120 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:26.554 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.671+0000 7f4571916640 10 monclient: get_monmap_and_config 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.671+0000 7f4571916640 10 monclient: build_initial_monmap 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.671+0000 7f4571916640 10 monclient: monmap: 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:40:26.673795+0000 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:40:26.673795+0000 2024-03-25T17:40:26.673 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:26.674 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:26.674 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:26.674 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:40:26.674 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:40:26.674 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:26.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.672+0000 7f4571916640 1 Processor -- start 2024-03-25T17:40:26.675 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.672+0000 7f4571916640 1 -- start start 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.672+0000 7f4571916640 10 monclient: init 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 10 monclient(hunting): picked mon.noname-c con 0x7f456c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 10 monclient(hunting): picked mon.noname-b con 0x7f456c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 10 monclient(hunting): picked mon.noname-a con 0x7f456c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c142520 con 0x7f456c1430c0 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c0b28a0 con 0x7f456c0a98f0 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c0b2a30 con 0x7f456c13f710 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:26.676 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.673+0000 7f4571916640 10 monclient(hunting): authenticate will time out at 5979.937345s 2024-03-25T17:40:26.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.688+0000 7f456a7fc640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f456c13f710 0x7f456c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:56872/0 (socket says 172.21.15.134:56872) 2024-03-25T17:40:26.689 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.688+0000 7f456a7fc640 1 -- 172.21.15.134:0/3780879503 learned_addr learned my addr 172.21.15.134:0/3780879503 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:26.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.704+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952471037 0 0) 0x7f456c0b2a30 con 0x7f456c13f710 2024-03-25T17:40:26.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.704+0000 7f4569ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:26.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.704+0000 7f4569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:26.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.704+0000 7f4569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:26.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.704+0000 7f4569ffb640 10 monclient(hunting): my global_id is 34685 2024-03-25T17:40:26.706 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.704+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4554003520 con 0x7f456c13f710 2024-03-25T17:40:26.707 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.706+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 364766384 0 0) 0x7f456c142520 con 0x7f456c1430c0 2024-03-25T17:40:26.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.706+0000 7f4569ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:26.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.706+0000 7f4569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:26.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.706+0000 7f4569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:26.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.706+0000 7f4569ffb640 10 monclient(hunting): my global_id is 4704 2024-03-25T17:40:26.708 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.706+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f456c0b2a30 con 0x7f456c1430c0 2024-03-25T17:40:26.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.709+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38049708 0 0) 0x7f456c0b28a0 con 0x7f456c0a98f0 2024-03-25T17:40:26.710 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.709+0000 7f4569ffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:26.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.709+0000 7f4569ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:26.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.709+0000 7f4569ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:26.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.709+0000 7f4569ffb640 10 monclient(hunting): my global_id is 4732 2024-03-25T17:40:26.711 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.709+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f456c142520 con 0x7f456c0a98f0 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3640513937 0 0) 0x7f4554003520 con 0x7f456c13f710 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 >> v1:172.21.15.159:6789/0 conn(0x7f456c0a98f0 legacy=0x7f456c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 >> v1:172.21.15.134:6789/0 conn(0x7f456c1430c0 legacy=0x7f456c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 10 monclient: _finish_hunting 0 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 1 monclient: found mon.noname-b 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.134:6790/0 2024-03-25T17:40:26.715 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c0b44c0 con 0x7f456c13f710 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 10 monclient: _finish_auth 0 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 10 monclient: _check_auth_tickets 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4560001690 con 0x7f456c13f710 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4569ffb640 10 monclient: dump: 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:26.716 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:26.717 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:26.717 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:26.717 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:26.717 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:26.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4571916640 5 monclient: authenticate success, global_id 34685 2024-03-25T17:40:26.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.713+0000 7f4571916640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:26.736 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 <== mon.2 v1:172.21.15.134:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45600017f0 con 0x7f456c13f710 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4569ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4569ffb640 1 -- 172.21.15.134:0/3780879503 <== mon.2 v1:172.21.15.134:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4560001c40 con 0x7f456c13f710 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4571916640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4571916640 4 set_mon_vals no callback set 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4569ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.735+0000 7f4569ffb640 10 monclient: dump: 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:26.737 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:26.738 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:26.738 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:26.738 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:26.738 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.736+0000 7f4571916640 10 monclient: shutdown 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.736+0000 7f4571916640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.736+0000 7f4571916640 1 -- 172.21.15.134:0/3780879503 >> v1:172.21.15.134:6790/0 conn(0x7f456c13f710 legacy=0x7f456c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.736+0000 7f4571916640 1 -- 172.21.15.134:0/3780879503 shutdown_connections 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.736+0000 7f4571916640 1 -- 172.21.15.134:0/3780879503 >> 172.21.15.134:0/3780879503 conn(0x7f456c0b7ea0 msgr2=0x7f456c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.736+0000 7f4571916640 1 -- 172.21.15.134:0/3780879503 shutdown_connections 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.736+0000 7f4571916640 1 -- 172.21.15.134:0/3780879503 wait complete. 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.737+0000 7f4571916640 10 monclient: build_initial_monmap 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.737+0000 7f4571916640 10 monclient: monmap: 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:26.739 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.738+0000 7f4571916640 1 librados: starting msgr at 2024-03-25T17:40:26.740 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.738+0000 7f4571916640 1 librados: starting objecter 2024-03-25T17:40:26.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.738+0000 7f4571916640 1 Processor -- start 2024-03-25T17:40:26.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.738+0000 7f4571916640 1 -- start start 2024-03-25T17:40:26.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.738+0000 7f4571916640 1 librados: setting wanted keys 2024-03-25T17:40:26.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.738+0000 7f4571916640 1 librados: calling monclient init 2024-03-25T17:40:26.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.738+0000 7f4571916640 10 monclient: init 2024-03-25T17:40:26.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:26.741 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 10 monclient(hunting): picked mon.noname-b con 0x7f456c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 10 monclient(hunting): picked mon.noname-c con 0x7f456c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 10 monclient(hunting): picked mon.noname-a con 0x7f456c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c1396c0 con 0x7f456c1430c0 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c1db830 con 0x7f456c0a98f0 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456c1399a0 con 0x7f456c13f710 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:26.742 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.739+0000 7f4571916640 10 monclient(hunting): authenticate will time out at 5980.003218s 2024-03-25T17:40:26.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.746+0000 7f456b7fe640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f456c1430c0 0x7f456c137900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:48700/0 (socket says 172.21.15.134:48700) 2024-03-25T17:40:26.748 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.746+0000 7f456b7fe640 1 -- 172.21.15.134:0/2195782293 learned_addr learned my addr 172.21.15.134:0/2195782293 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:26.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.749+0000 7f4553fff640 10 client.?.objecter ms_handle_connect 0x7f456c1430c0 2024-03-25T17:40:26.750 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.749+0000 7f4553fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:26.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.758+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761297599 0 0) 0x7f456c1396c0 con 0x7f456c1430c0 2024-03-25T17:40:26.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.758+0000 7f4553fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:26.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.758+0000 7f4553fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:26.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.758+0000 7f4553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:26.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.758+0000 7f4553fff640 10 monclient(hunting): my global_id is 4707 2024-03-25T17:40:26.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.758+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4558004eb0 con 0x7f456c1430c0 2024-03-25T17:40:26.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.764+0000 7f4553fff640 10 client.?.objecter ms_handle_connect 0x7f456c0a98f0 2024-03-25T17:40:26.765 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.764+0000 7f4553fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:26.772 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.771+0000 7f4553fff640 10 client.?.objecter ms_handle_connect 0x7f456c13f710 2024-03-25T17:40:26.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.771+0000 7f4553fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:26.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.771+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563181708 0 0) 0x7f4558004eb0 con 0x7f456c1430c0 2024-03-25T17:40:26.773 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.771+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f456c1396c0 con 0x7f456c1430c0 2024-03-25T17:40:26.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.775+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4564001db0 con 0x7f456c1430c0 2024-03-25T17:40:26.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.775+0000 7f4553fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:26.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.775+0000 7f4553fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:26.776 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.775+0000 7f4553fff640 10 monclient(hunting): dump: 2024-03-25T17:40:26.776 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:26.776 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:26.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.775+0000 7f4553fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:26.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.777+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612818005 0 0) 0x7f456c1db830 con 0x7f456c0a98f0 2024-03-25T17:40:26.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.777+0000 7f4553fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:26.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.777+0000 7f4553fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:26.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.777+0000 7f4553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:26.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.777+0000 7f4553fff640 10 monclient(hunting): my global_id is 4735 2024-03-25T17:40:26.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.777+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4558004eb0 con 0x7f456c0a98f0 2024-03-25T17:40:26.782 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.781+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381132264 0 0) 0x7f456c1399a0 con 0x7f456c13f710 2024-03-25T17:40:26.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.781+0000 7f4553fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:26.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.781+0000 7f4553fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:26.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.781+0000 7f4553fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:26.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.781+0000 7f4553fff640 10 monclient(hunting): my global_id is 34688 2024-03-25T17:40:26.783 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.781+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f456c1db830 con 0x7f456c13f710 2024-03-25T17:40:26.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.792+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3670448341 0 0) 0x7f456c1db830 con 0x7f456c13f710 2024-03-25T17:40:26.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.792+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f456c1399a0 con 0x7f456c13f710 2024-03-25T17:40:26.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.792+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4560001b00 con 0x7f456c13f710 2024-03-25T17:40:26.794 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.792+0000 7f4553fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.792+0000 7f4553fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.792+0000 7f4553fff640 10 monclient(hunting): dump: 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:26.795 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3613473994 0 0) 0x7f456c1396c0 con 0x7f456c1430c0 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 >> v1:172.21.15.134:6790/0 conn(0x7f456c13f710 legacy=0x7f456c0b0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 >> v1:172.21.15.159:6789/0 conn(0x7f456c0a98f0 legacy=0x7f456c1dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 1 monclient: found mon.a 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f456c1db380 con 0x7f456c1430c0 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 10 monclient: _finish_auth 0 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:26.797 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4571916640 5 monclient: authenticate success, global_id 4707 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4571916640 10 monclient: _renew_subs 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4571916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.793+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456c0b0840 con 0x7f456c1430c0 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.794+0000 7f4571916640 10 client.4707.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.794+0000 7f4571916640 10 monclient: _renew_subs 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.794+0000 7f4571916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.794+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f456c138010 con 0x7f456c1430c0 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.794+0000 7f4571916640 1 librados: init done 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.795+0000 7f4571916640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.795+0000 7f4571916640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.795+0000 7f4571916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:26.798 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:26.795+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456c0627b0 con 0x7f456c1430c0 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4564001f10 con 0x7f456c1430c0 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f4553fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f4564003dc0 con 0x7f456c1430c0 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f4553fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f4553fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f4553fff640 10 monclient: dump: 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:27.101 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:27.102 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:27.102 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:27.102 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:27.102 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:27.102 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:27.102 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f45697fa640 4 set_mon_vals no callback set 2024-03-25T17:40:27.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.099+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f4564003fe0 con 0x7f456c1430c0 2024-03-25T17:40:27.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.100+0000 7f456affd640 10 monclient: get_auth_request con 0x7f455806dfd0 auth_method 0 2024-03-25T17:40:27.103 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.100+0000 7f4553fff640 10 client.4707.objecter ms_handle_connect 0x7f455806dfd0 2024-03-25T17:40:27.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f4564054b40 con 0x7f456c1430c0 2024-03-25T17:40:27.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 10 client.4707.objecter ms_dispatch 0x7f456c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:40:27.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 3 client.4707.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:40:27.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 3 client.4707.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:40:27.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 20 client.4707.objecter dump_active .. 0 homeless 2024-03-25T17:40:27.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4564050650 con 0x7f456c1430c0 2024-03-25T17:40:27.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:27.108 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.106+0000 7f4553fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:27.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.314+0000 7f4571916640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:27.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.314+0000 7f4571916640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:27.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.314+0000 7f4571916640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.134:6789/0 2024-03-25T17:40:27.316 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.314+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 --> v1:172.21.15.134:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f456c074cc0 con 0x7f456c1430c0 2024-03-25T17:40:27.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.389+0000 7f4553fff640 1 -- 172.21.15.134:0/2195782293 <== mon.0 v1:172.21.15.134:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f4564052580 con 0x7f456c1430c0 2024-03-25T17:40:27.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.389+0000 7f4553fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:27.391 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.389+0000 7f4553fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:27.391 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:40:27.391 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:27.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 10 librados: watch_flush enter 2024-03-25T17:40:27.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 10 librados: watch_flush exit 2024-03-25T17:40:27.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 20 client.4707.objecter shutdown clearing up homeless session... 2024-03-25T17:40:27.393 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 10 client.4707.objecter successfully canceled tick 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 >> v1:172.21.15.134:6800/610837766 conn(0x7f455806dfd0 legacy=0x7f4558070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 10 monclient: shutdown 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 >> v1:172.21.15.134:6789/0 conn(0x7f456c1430c0 legacy=0x7f456c137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 shutdown_connections 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.391+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 >> 172.21.15.134:0/2195782293 conn(0x7f456c0b7ea0 msgr2=0x7f456c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.392+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 shutdown_connections 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.392+0000 7f4571916640 1 -- 172.21.15.134:0/2195782293 wait complete. 2024-03-25T17:40:27.394 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:27.392+0000 7f4571916640 1 librados: shutdown 2024-03-25T17:40:27.404 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:34.211 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:34.329 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.329+0000 7f816f523640 10 monclient: get_monmap_and_config 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.329+0000 7f816f523640 10 monclient: build_initial_monmap 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.329+0000 7f816f523640 10 monclient: monmap: 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:40:34.330341+0000 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:40:34.330341+0000 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:40:34.330 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:34.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.330+0000 7f816f523640 1 Processor -- start 2024-03-25T17:40:34.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.330+0000 7f816f523640 1 -- start start 2024-03-25T17:40:34.331 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.330+0000 7f816f523640 10 monclient: init 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 10 monclient(hunting): picked mon.noname-c con 0x7f81680a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 10 monclient(hunting): picked mon.noname-b con 0x7f816813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 10 monclient(hunting): picked mon.noname-a con 0x7f81681430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8168142520 con 0x7f81681430c0 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81680b28a0 con 0x7f81680a98f0 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81680b2a30 con 0x7f816813f710 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:34.332 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.331+0000 7f816f523640 10 monclient(hunting): authenticate will time out at 5987.594318s 2024-03-25T17:40:34.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.336+0000 7f816dac0640 1 --1- >> v1:172.21.15.134:6789/0 conn(0x7f81681430c0 0x7f816814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6789/0 says I am v1:172.21.15.134:49270/0 (socket says 172.21.15.134:49270) 2024-03-25T17:40:34.336 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.336+0000 7f816dac0640 1 -- 172.21.15.134:0/673868621 learned_addr learned my addr 172.21.15.134:0/673868621 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:34.349 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.349+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18168504 0 0) 0x7f8168142520 con 0x7f81681430c0 2024-03-25T17:40:34.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.349+0000 7f815ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:34.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.349+0000 7f815ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:34.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.349+0000 7f815ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:34.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.349+0000 7f815ffff640 10 monclient(hunting): my global_id is 4710 2024-03-25T17:40:34.350 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.349+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8154003520 con 0x7f81681430c0 2024-03-25T17:40:34.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.355+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475429320 0 0) 0x7f81680b2a30 con 0x7f816813f710 2024-03-25T17:40:34.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.355+0000 7f815ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:34.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.355+0000 7f815ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:34.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.355+0000 7f815ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:34.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.355+0000 7f815ffff640 10 monclient(hunting): my global_id is 34691 2024-03-25T17:40:34.356 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.355+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8168142520 con 0x7f816813f710 2024-03-25T17:40:34.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.357+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482681523 0 0) 0x7f81680b28a0 con 0x7f81680a98f0 2024-03-25T17:40:34.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.357+0000 7f815ffff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:34.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.357+0000 7f815ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:34.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.357+0000 7f815ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:34.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.357+0000 7f815ffff640 10 monclient(hunting): my global_id is 4738 2024-03-25T17:40:34.357 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.357+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81680b2a30 con 0x7f81680a98f0 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 <== mon.0 v1:172.21.15.134:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1149614626 0 0) 0x7f8154003520 con 0x7f81681430c0 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 >> v1:172.21.15.134:6790/0 conn(0x7f816813f710 legacy=0x7f81681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 >> v1:172.21.15.159:6789/0 conn(0x7f81680a98f0 legacy=0x7f81680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 1 monclient: found mon.noname-a 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.134:6789/0 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 --> v1:172.21.15.134:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680b44c0 con 0x7f81681430c0 2024-03-25T17:40:34.370 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 10 monclient: _finish_auth 0 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 <== mon.0 v1:172.21.15.134:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8160001a10 con 0x7f81681430c0 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f815ffff640 10 monclient: dump: 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:34.371 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:34.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f816f523640 5 monclient: authenticate success, global_id 4710 2024-03-25T17:40:34.372 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.369+0000 7f816f523640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 <== mon.0 v1:172.21.15.134:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8160001b70 con 0x7f81681430c0 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f815ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f815ffff640 1 -- 172.21.15.134:0/673868621 <== mon.0 v1:172.21.15.134:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8160001f80 con 0x7f81681430c0 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f815ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f815ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f815ffff640 10 monclient: dump: 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:34.382 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f816f523640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:34.383 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.381+0000 7f816f523640 4 set_mon_vals no callback set 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.382+0000 7f816f523640 10 monclient: shutdown 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.382+0000 7f816f523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.382+0000 7f816f523640 1 -- 172.21.15.134:0/673868621 >> v1:172.21.15.134:6789/0 conn(0x7f81681430c0 legacy=0x7f816814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.382+0000 7f816f523640 1 -- 172.21.15.134:0/673868621 shutdown_connections 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.382+0000 7f816f523640 1 -- 172.21.15.134:0/673868621 >> 172.21.15.134:0/673868621 conn(0x7f81680b7ea0 msgr2=0x7f81680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.382+0000 7f816f523640 1 -- 172.21.15.134:0/673868621 shutdown_connections 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.382+0000 7f816f523640 1 -- 172.21.15.134:0/673868621 wait complete. 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 10 monclient: build_initial_monmap 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 10 monclient: monmap: 2024-03-25T17:40:34.385 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 1 librados: starting msgr at 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 1 librados: starting objecter 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 1 Processor -- start 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 1 -- start start 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 1 librados: setting wanted keys 2024-03-25T17:40:34.386 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 1 librados: calling monclient init 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.383+0000 7f816f523640 10 monclient: init 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 10 monclient(hunting): picked mon.noname-a con 0x7f81680a98f0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 10 monclient(hunting): picked mon.noname-c con 0x7f816813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 10 monclient(hunting): picked mon.noname-b con 0x7f81681430c0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8168136c50 con 0x7f81680a98f0 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8168136ec0 con 0x7f81681430c0 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81681317f0 con 0x7f816813f710 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:34.387 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.384+0000 7f816f523640 10 monclient(hunting): authenticate will time out at 5987.647106s 2024-03-25T17:40:34.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.395+0000 7f816cabe640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f816813f710 0x7f81681dac70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:56478/0 (socket says 172.21.15.134:56478) 2024-03-25T17:40:34.395 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.395+0000 7f816cabe640 1 -- 172.21.15.134:0/3575471264 learned_addr learned my addr 172.21.15.134:0/3575471264 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:34.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.396+0000 7f815dffb640 10 client.?.objecter ms_handle_connect 0x7f816813f710 2024-03-25T17:40:34.397 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.397+0000 7f815dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:34.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.406+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270932974 0 0) 0x7f81681317f0 con 0x7f816813f710 2024-03-25T17:40:34.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.406+0000 7f815dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:34.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.406+0000 7f815dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:34.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.406+0000 7f815dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:34.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.406+0000 7f815dffb640 10 monclient(hunting): my global_id is 34694 2024-03-25T17:40:34.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.406+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f814c004eb0 con 0x7f816813f710 2024-03-25T17:40:34.407 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.407+0000 7f815dffb640 10 client.?.objecter ms_handle_connect 0x7f81680a98f0 2024-03-25T17:40:34.408 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.407+0000 7f815dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:34.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.412+0000 7f815dffb640 10 client.?.objecter ms_handle_connect 0x7f81681430c0 2024-03-25T17:40:34.412 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.412+0000 7f815dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:34.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.431+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184635574 0 0) 0x7f8168136ec0 con 0x7f81681430c0 2024-03-25T17:40:34.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.431+0000 7f815dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:34.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.431+0000 7f815dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:34.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.431+0000 7f815dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:34.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.431+0000 7f815dffb640 10 monclient(hunting): my global_id is 4741 2024-03-25T17:40:34.432 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.431+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81681317f0 con 0x7f81681430c0 2024-03-25T17:40:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.434+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982729550 0 0) 0x7f8168136c50 con 0x7f81680a98f0 2024-03-25T17:40:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.434+0000 7f815dffb640 10 monclient(hunting): _negotiate 2024-03-25T17:40:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.434+0000 7f815dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.434+0000 7f815dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.434+0000 7f815dffb640 10 monclient(hunting): my global_id is 4713 2024-03-25T17:40:34.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.434+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8168136ec0 con 0x7f81680a98f0 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.436+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1969944474 0 0) 0x7f814c004eb0 con 0x7f816813f710 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.436+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8168136c50 con 0x7f816813f710 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.436+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8158001a30 con 0x7f816813f710 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.436+0000 7f815dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.436+0000 7f815dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.436+0000 7f815dffb640 10 monclient(hunting): dump: 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.437 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:34.438 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:34.438 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:34.438 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:34.438 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:34.438 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:34.438 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.436+0000 7f815dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.449+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2652551073 0 0) 0x7f81681317f0 con 0x7f81681430c0 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.450+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f814c004eb0 con 0x7f81681430c0 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.450+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8160001e80 con 0x7f81681430c0 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.450+0000 7f815dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.450+0000 7f815dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.450+0000 7f815dffb640 10 monclient(hunting): dump: 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:34.450 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:34.451 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 617394532 0 0) 0x7f8168136c50 con 0x7f816813f710 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 >> v1:172.21.15.159:6789/0 conn(0x7f81681430c0 legacy=0x7f81681db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 >> v1:172.21.15.134:6789/0 conn(0x7f81680a98f0 legacy=0x7f81681310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 10 monclient: _finish_hunting 0 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 1 monclient: found mon.c 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:34.456 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8168131980 con 0x7f816813f710 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 10 monclient: _finish_auth 0 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 10 monclient: _check_auth_tickets 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f815dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f816f523640 5 monclient: authenticate success, global_id 34694 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f816f523640 10 monclient: _renew_subs 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f816f523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.455+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8168135070 con 0x7f816813f710 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.456+0000 7f816f523640 10 client.34694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.456+0000 7f816f523640 10 monclient: _renew_subs 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.456+0000 7f816f523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.456+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81680b7870 con 0x7f816813f710 2024-03-25T17:40:34.457 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.456+0000 7f816f523640 1 librados: init done 2024-03-25T17:40:34.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.457+0000 7f816f523640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:34.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.457+0000 7f816f523640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:34.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.457+0000 7f816f523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:34.458 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.457+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8168135c70 con 0x7f816813f710 2024-03-25T17:40:34.777 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8158001c70 con 0x7f816813f710 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f8158003a60 con 0x7f816813f710 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815dffb640 10 monclient: dump: 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:34.778 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:34.779 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:34.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815f7fe640 4 set_mon_vals no callback set 2024-03-25T17:40:34.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.777+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f8158003c80 con 0x7f816813f710 2024-03-25T17:40:34.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.778+0000 7f816d2bf640 10 monclient: get_auth_request con 0x7f814c06df70 auth_method 0 2024-03-25T17:40:34.779 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.778+0000 7f815dffb640 10 client.34694.objecter ms_handle_connect 0x7f814c06df70 2024-03-25T17:40:34.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f81580547f0 con 0x7f816813f710 2024-03-25T17:40:34.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 10 client.34694.objecter ms_dispatch 0x7f8168000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:40:34.786 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 3 client.34694.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:40:34.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 3 client.34694.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:40:34.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 20 client.34694.objecter dump_active .. 0 homeless 2024-03-25T17:40:34.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8158050300 con 0x7f816813f710 2024-03-25T17:40:34.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:34.787 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:34.786+0000 7f815dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:35.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.017+0000 7f816f523640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:35.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.017+0000 7f816f523640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:35.018 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.017+0000 7f816f523640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.134:6790/0 2024-03-25T17:40:35.019 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.017+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 --> v1:172.21.15.134:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81680627b0 con 0x7f816813f710 2024-03-25T17:40:35.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.060+0000 7f815dffb640 1 -- 172.21.15.134:0/3575471264 <== mon.2 v1:172.21.15.134:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f8158052230 con 0x7f816813f710 2024-03-25T17:40:35.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.060+0000 7f815dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:35.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.060+0000 7f815dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:35.061 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:40:35.061 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 10 librados: watch_flush enter 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 10 librados: watch_flush exit 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 20 client.34694.objecter shutdown clearing up homeless session... 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 10 client.34694.objecter successfully canceled tick 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 >> v1:172.21.15.134:6800/610837766 conn(0x7f814c06df70 legacy=0x7f814c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 10 monclient: shutdown 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 >> v1:172.21.15.134:6790/0 conn(0x7f816813f710 legacy=0x7f81681dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 shutdown_connections 2024-03-25T17:40:35.064 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 >> 172.21.15.134:0/3575471264 conn(0x7f81680b7ea0 msgr2=0x7f816813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:35.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 shutdown_connections 2024-03-25T17:40:35.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 1 -- 172.21.15.134:0/3575471264 wait complete. 2024-03-25T17:40:35.065 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:35.063+0000 7f816f523640 1 librados: shutdown 2024-03-25T17:40:35.075 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:36.335 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:41.352 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.379 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:41.408 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.382 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:41.411 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.389 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:41.406 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.393 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:11:41.422 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.428 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:11:41.444 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.434 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:41.451 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.445 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:41.461 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:36.471 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:11:41.488 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:41.882 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.001+0000 7f912214d640 10 monclient: get_monmap_and_config 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.001+0000 7f912214d640 10 monclient: build_initial_monmap 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.001+0000 7f912214d640 10 monclient: monmap: 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:40:42.002463+0000 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:40:42.002463+0000 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:40:42.002 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:40:42.003 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.002+0000 7f912214d640 1 Processor -- start 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.002+0000 7f912214d640 1 -- start start 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.002+0000 7f912214d640 10 monclient: init 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 10 monclient(hunting): picked mon.noname-c con 0x7f911c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 10 monclient(hunting): picked mon.noname-b con 0x7f911c13f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 10 monclient(hunting): picked mon.noname-a con 0x7f911c1430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c142520 con 0x7f911c1430c0 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c0b28a0 con 0x7f911c0a98f0 2024-03-25T17:40:42.004 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c0b2a30 con 0x7f911c13f710 2024-03-25T17:40:42.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:42.005 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.003+0000 7f912214d640 10 monclient(hunting): authenticate will time out at 5995.265997s 2024-03-25T17:40:42.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.015+0000 7f911affd640 1 --1- >> v1:172.21.15.134:6790/0 conn(0x7f911c13f710 0x7f911c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.134:6790/0 says I am v1:172.21.15.134:35876/0 (socket says 172.21.15.134:35876) 2024-03-25T17:40:42.015 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.015+0000 7f911affd640 1 -- 172.21.15.134:0/2372842297 learned_addr learned my addr 172.21.15.134:0/2372842297 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:42.030 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.030+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3940906223 0 0) 0x7f911c0b2a30 con 0x7f911c13f710 2024-03-25T17:40:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.030+0000 7f911a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.030+0000 7f911a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.030+0000 7f911a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.030+0000 7f911a7fc640 10 monclient(hunting): my global_id is 34697 2024-03-25T17:40:42.031 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.030+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9104003520 con 0x7f911c13f710 2024-03-25T17:40:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.032+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568772229 0 0) 0x7f911c0b28a0 con 0x7f911c0a98f0 2024-03-25T17:40:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.032+0000 7f911a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.032+0000 7f911a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:42.032 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.032+0000 7f911a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.032+0000 7f911a7fc640 10 monclient(hunting): my global_id is 4744 2024-03-25T17:40:42.033 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.032+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f911c0b2a30 con 0x7f911c0a98f0 2024-03-25T17:40:42.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.052+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429134020 0 0) 0x7f911c142520 con 0x7f911c1430c0 2024-03-25T17:40:42.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.052+0000 7f911a7fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:42.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.052+0000 7f911a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:42.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.052+0000 7f911a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:42.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.052+0000 7f911a7fc640 10 monclient(hunting): my global_id is 4716 2024-03-25T17:40:42.052 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.052+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f911c0b28a0 con 0x7f911c1430c0 2024-03-25T17:40:42.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1737372203 0 0) 0x7f911c0b2a30 con 0x7f911c0a98f0 2024-03-25T17:40:42.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 >> v1:172.21.15.134:6790/0 conn(0x7f911c13f710 legacy=0x7f911c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:42.061 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 >> v1:172.21.15.134:6789/0 conn(0x7f911c1430c0 legacy=0x7f911c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 1 monclient: found mon.noname-c 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c0b44c0 con 0x7f911c0a98f0 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f910c001670 con 0x7f911c0a98f0 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f911a7fc640 10 monclient: dump: 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:42.062 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f912214d640 5 monclient: authenticate success, global_id 4744 2024-03-25T17:40:42.063 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.060+0000 7f912214d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f910c0017d0 con 0x7f911c0a98f0 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f911a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f911a7fc640 1 -- 172.21.15.134:0/2372842297 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f910c001be0 con 0x7f911c0a98f0 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f912214d640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f912214d640 4 set_mon_vals no callback set 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f911a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f911a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.076+0000 7f911a7fc640 10 monclient: dump: 2024-03-25T17:40:42.077 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:42.078 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:42.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.077+0000 7f912214d640 10 monclient: shutdown 2024-03-25T17:40:42.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.077+0000 7f912214d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:42.080 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.077+0000 7f912214d640 1 -- 172.21.15.134:0/2372842297 >> v1:172.21.15.159:6789/0 conn(0x7f911c0a98f0 legacy=0x7f911c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.077+0000 7f912214d640 1 -- 172.21.15.134:0/2372842297 shutdown_connections 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.077+0000 7f912214d640 1 -- 172.21.15.134:0/2372842297 >> 172.21.15.134:0/2372842297 conn(0x7f911c0b7ea0 msgr2=0x7f911c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.077+0000 7f912214d640 1 -- 172.21.15.134:0/2372842297 shutdown_connections 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.077+0000 7f912214d640 1 -- 172.21.15.134:0/2372842297 wait complete. 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.078+0000 7f912214d640 10 monclient: build_initial_monmap 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.078+0000 7f912214d640 10 monclient: monmap: 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:40:42.081 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 librados: starting msgr at 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 librados: starting objecter 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 Processor -- start 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 -- start start 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 librados: setting wanted keys 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 librados: calling monclient init 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 10 monclient: init 2024-03-25T17:40:42.082 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 10 monclient(hunting): picked mon.noname-b con 0x7f911c0a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 10 monclient(hunting): picked mon.noname-a con 0x7f911c13f710 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 10 monclient(hunting): picked mon.noname-c con 0x7f911c1430c0 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c13b180 con 0x7f911c13f710 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.079+0000 7f912214d640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c1db890 con 0x7f911c0a98f0 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.080+0000 7f912214d640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911c0b7b90 con 0x7f911c1430c0 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.080+0000 7f912214d640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:42.083 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.080+0000 7f912214d640 10 monclient(hunting): authenticate will time out at 5995.342805s 2024-03-25T17:40:42.085 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.085+0000 7f911b7fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f911c0a98f0 0x7f911c0b7480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:46716/0 (socket says 172.21.15.134:46716) 2024-03-25T17:40:42.086 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.085+0000 7f911b7fe640 1 -- 172.21.15.134:0/1737434088 learned_addr learned my addr 172.21.15.134:0/1737434088 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:42.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.086+0000 7f9103fff640 10 client.?.objecter ms_handle_connect 0x7f911c0a98f0 2024-03-25T17:40:42.087 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.086+0000 7f9103fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:42.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.088+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515628213 0 0) 0x7f911c1db890 con 0x7f911c0a98f0 2024-03-25T17:40:42.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.088+0000 7f9103fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:42.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.088+0000 7f9103fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:42.088 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.088+0000 7f9103fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:42.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.088+0000 7f9103fff640 10 monclient(hunting): my global_id is 4747 2024-03-25T17:40:42.089 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.088+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9108004eb0 con 0x7f911c0a98f0 2024-03-25T17:40:42.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.097+0000 7f9103fff640 10 client.?.objecter ms_handle_connect 0x7f911c1430c0 2024-03-25T17:40:42.098 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.097+0000 7f9103fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:42.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.098+0000 7f9103fff640 10 client.?.objecter ms_handle_connect 0x7f911c13f710 2024-03-25T17:40:42.099 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.098+0000 7f9103fff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:42.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.105+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1921915275 0 0) 0x7f9108004eb0 con 0x7f911c0a98f0 2024-03-25T17:40:42.105 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.105+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f911c1db890 con 0x7f911c0a98f0 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.105+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f910c001be0 con 0x7f911c0a98f0 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.105+0000 7f9103fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.105+0000 7f9103fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.105+0000 7f9103fff640 10 monclient(hunting): dump: 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:42.106 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:42.107 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:42.107 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:42.107 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:42.107 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.105+0000 7f9103fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:42.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.117+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787046681 0 0) 0x7f911c0b7b90 con 0x7f911c1430c0 2024-03-25T17:40:42.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.117+0000 7f9103fff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:42.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.117+0000 7f9103fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:42.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.117+0000 7f9103fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:42.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.117+0000 7f9103fff640 10 monclient(hunting): my global_id is 34700 2024-03-25T17:40:42.118 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.117+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9108004eb0 con 0x7f911c1430c0 2024-03-25T17:40:42.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2360385287 0 0) 0x7f911c1db890 con 0x7f911c0a98f0 2024-03-25T17:40:42.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 >> v1:172.21.15.134:6790/0 conn(0x7f911c1430c0 legacy=0x7f911c1db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:42.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 >> v1:172.21.15.134:6789/0 conn(0x7f911c13f710 legacy=0x7f911c1daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:42.136 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 1 monclient: found mon.b 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c13b3f0 con 0x7f911c0a98f0 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 10 monclient: _finish_auth 0 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f9103fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f912214d640 5 monclient: authenticate success, global_id 4747 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f912214d640 10 monclient: _renew_subs 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f912214d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.135+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f911c13a1b0 con 0x7f911c0a98f0 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.136+0000 7f912214d640 10 client.4747.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.136+0000 7f912214d640 10 monclient: _renew_subs 2024-03-25T17:40:42.137 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.136+0000 7f912214d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:42.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.136+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f911c13b9f0 con 0x7f911c0a98f0 2024-03-25T17:40:42.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.136+0000 7f912214d640 1 librados: init done 2024-03-25T17:40:42.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.137+0000 7f912214d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:42.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.137+0000 7f912214d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:42.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.137+0000 7f912214d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:42.138 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.137+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911c0b3ef0 con 0x7f911c0a98f0 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.434+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f910c0011f0 con 0x7f911c0a98f0 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.434+0000 7f9103fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.434+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f910c003a40 con 0x7f911c0a98f0 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.434+0000 7f9103fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.434+0000 7f9103fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.434+0000 7f9103fff640 10 monclient: dump: 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:42.435 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:42.436 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.434+0000 7f9119ffb640 4 set_mon_vals no callback set 2024-03-25T17:40:42.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.435+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f910c003c20 con 0x7f911c0a98f0 2024-03-25T17:40:42.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.436+0000 7f911affd640 10 monclient: get_auth_request con 0x7f910806dd40 auth_method 0 2024-03-25T17:40:42.437 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.436+0000 7f9103fff640 10 client.4747.objecter ms_handle_connect 0x7f910806dd40 2024-03-25T17:40:42.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.441+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f910c0548e0 con 0x7f911c0a98f0 2024-03-25T17:40:42.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.441+0000 7f9103fff640 10 client.4747.objecter ms_dispatch 0x7f911c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:40:42.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.441+0000 7f9103fff640 3 client.4747.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:40:42.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.441+0000 7f9103fff640 3 client.4747.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:40:42.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.441+0000 7f9103fff640 20 client.4747.objecter dump_active .. 0 homeless 2024-03-25T17:40:42.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.442+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f910c050220 con 0x7f911c0a98f0 2024-03-25T17:40:42.442 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.442+0000 7f9103fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:42.443 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.442+0000 7f9103fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:42.694 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.694+0000 7f912214d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:42.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.694+0000 7f912214d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:42.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.694+0000 7f912214d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:42.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.694+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f911c0627b0 con 0x7f911c0a98f0 2024-03-25T17:40:42.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.724+0000 7f9103fff640 1 -- 172.21.15.134:0/1737434088 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f910c052320 con 0x7f911c0a98f0 2024-03-25T17:40:42.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.724+0000 7f9103fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:42.725 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.724+0000 7f9103fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:42.725 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:40:42.726 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:42.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 10 librados: watch_flush enter 2024-03-25T17:40:42.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 10 librados: watch_flush exit 2024-03-25T17:40:42.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 20 client.4747.objecter shutdown clearing up homeless session... 2024-03-25T17:40:42.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 10 client.4747.objecter successfully canceled tick 2024-03-25T17:40:42.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 >> v1:172.21.15.134:6800/610837766 conn(0x7f910806dd40 legacy=0x7f9108070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:42.728 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 10 monclient: shutdown 2024-03-25T17:40:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 >> v1:172.21.15.159:6789/0 conn(0x7f911c0a98f0 legacy=0x7f911c0b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 shutdown_connections 2024-03-25T17:40:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 >> 172.21.15.134:0/1737434088 conn(0x7f911c0b7ea0 msgr2=0x7f911c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 shutdown_connections 2024-03-25T17:40:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 1 -- 172.21.15.134:0/1737434088 wait complete. 2024-03-25T17:40:42.729 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:42.727+0000 7f912214d640 1 librados: shutdown 2024-03-25T17:40:42.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:49.547 DEBUG:teuthology.orchestra.run.smithi134:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T17:40:49.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.664+0000 7f170daff640 10 monclient: get_monmap_and_config 2024-03-25T17:40:49.664 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.664+0000 7f170daff640 10 monclient: build_initial_monmap 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.664+0000 7f170daff640 10 monclient: monmap: 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:40:49.665119+0000 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:40:49.665119+0000 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-c 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-b 2024-03-25T17:40:49.665 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.664+0000 7f170daff640 1 Processor -- start 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 1 -- start start 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient: init 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient(hunting): picked mon.noname-c con 0x7f17080a98f0 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient(hunting): picked mon.noname-b con 0x7f170813f710 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient(hunting): picked mon.noname-a con 0x7f17081430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1708142520 con 0x7f17081430c0 2024-03-25T17:40:49.667 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17080b28a0 con 0x7f17080a98f0 2024-03-25T17:40:49.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17080b2a30 con 0x7f170813f710 2024-03-25T17:40:49.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:49.668 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.665+0000 7f170daff640 10 monclient(hunting): authenticate will time out at 6002.928746s 2024-03-25T17:40:49.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.671+0000 7f17077fe640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f17080a98f0 0x7f17080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:43106/0 (socket says 172.21.15.134:43106) 2024-03-25T17:40:49.671 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.671+0000 7f17077fe640 1 -- 172.21.15.134:0/1381868036 learned_addr learned my addr 172.21.15.134:0/1381868036 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:49.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.684+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139229029 0 0) 0x7f17080b28a0 con 0x7f17080a98f0 2024-03-25T17:40:49.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.684+0000 7f17067fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:49.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.684+0000 7f17067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:49.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.684+0000 7f17067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:49.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.684+0000 7f17067fc640 10 monclient(hunting): my global_id is 4750 2024-03-25T17:40:49.684 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.684+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f0003520 con 0x7f17080a98f0 2024-03-25T17:40:49.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.693+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102033233 0 0) 0x7f1708142520 con 0x7f17081430c0 2024-03-25T17:40:49.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.693+0000 7f17067fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:49.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.693+0000 7f17067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:49.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.693+0000 7f17067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:49.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.693+0000 7f17067fc640 10 monclient(hunting): my global_id is 4722 2024-03-25T17:40:49.693 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.693+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17080b28a0 con 0x7f17081430c0 2024-03-25T17:40:49.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.695+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077887358 0 0) 0x7f17080b2a30 con 0x7f170813f710 2024-03-25T17:40:49.695 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.695+0000 7f17067fc640 10 monclient(hunting): _negotiate 2024-03-25T17:40:49.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.695+0000 7f17067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:49.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.695+0000 7f17067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:49.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.695+0000 7f17067fc640 10 monclient(hunting): my global_id is 34703 2024-03-25T17:40:49.696 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.695+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1708142520 con 0x7f170813f710 2024-03-25T17:40:49.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 54959623 0 0) 0x7f16f0003520 con 0x7f17080a98f0 2024-03-25T17:40:49.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 >> v1:172.21.15.134:6790/0 conn(0x7f170813f710 legacy=0x7f17081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:49.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 >> v1:172.21.15.134:6789/0 conn(0x7f17081430c0 legacy=0x7f170814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:49.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 10 monclient: _finish_hunting 0 2024-03-25T17:40:49.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 1 monclient: found mon.noname-c 2024-03-25T17:40:49.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:49.703 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.159:6789/0 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17080b44c0 con 0x7f17080a98f0 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 10 monclient: _finish_auth 0 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 10 monclient: _check_auth_tickets 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f16f8001690 con 0x7f17080a98f0 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f17067fc640 10 monclient: dump: 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:49.704 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:49.705 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:49.705 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:49.705 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:49.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f170daff640 5 monclient: authenticate success, global_id 4750 2024-03-25T17:40:49.705 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.702+0000 7f170daff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 <== mon.1 v1:172.21.15.159:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16f80017f0 con 0x7f17080a98f0 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f17067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f17067fc640 1 -- 172.21.15.134:0/1381868036 <== mon.1 v1:172.21.15.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f16f8001c00 con 0x7f17080a98f0 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f17067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f17067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f17067fc640 10 monclient: dump: 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:49.716 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f170daff640 10 monclient: get_monmap_and_config success 2024-03-25T17:40:49.717 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.715+0000 7f170daff640 4 set_mon_vals no callback set 2024-03-25T17:40:49.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.716+0000 7f170daff640 10 monclient: shutdown 2024-03-25T17:40:49.718 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.716+0000 7f170daff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.716+0000 7f170daff640 1 -- 172.21.15.134:0/1381868036 >> v1:172.21.15.159:6789/0 conn(0x7f17080a98f0 legacy=0x7f17080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.716+0000 7f170daff640 1 -- 172.21.15.134:0/1381868036 shutdown_connections 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.716+0000 7f170daff640 1 -- 172.21.15.134:0/1381868036 >> 172.21.15.134:0/1381868036 conn(0x7f17080b7ea0 msgr2=0x7f17080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.716+0000 7f170daff640 1 -- 172.21.15.134:0/1381868036 shutdown_connections 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.716+0000 7f170daff640 1 -- 172.21.15.134:0/1381868036 wait complete. 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.717+0000 7f170daff640 10 monclient: build_initial_monmap 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.717+0000 7f170daff640 10 monclient: monmap: 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:epoch 0 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:last_changed 0.000000 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:created 0.000000 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 0 (unknown) 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 1 2024-03-25T17:40:49.719 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.noname-a 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.noname-b 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.noname-c 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.717+0000 7f170daff640 1 librados: starting msgr at 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.717+0000 7f170daff640 1 librados: starting objecter 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.717+0000 7f170daff640 1 Processor -- start 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 1 -- start start 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 1 librados: setting wanted keys 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 1 librados: calling monclient init 2024-03-25T17:40:49.720 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 10 monclient: init 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 10 monclient: _reopen_session rank -1 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 10 monclient(hunting): picked mon.noname-c con 0x7f17080a98f0 addr v1:172.21.15.134:6790/0 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 10 monclient(hunting): picked mon.noname-b con 0x7f170813f710 addr v1:172.21.15.159:6789/0 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 10 monclient(hunting): picked mon.noname-a con 0x7f17081430c0 addr v1:172.21.15.134:6789/0 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 1 -- --> v1:172.21.15.134:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170813b0f0 con 0x7f17081430c0 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.718+0000 7f170daff640 1 -- --> v1:172.21.15.159:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170813b8f0 con 0x7f170813f710 2024-03-25T17:40:49.721 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.719+0000 7f170daff640 1 -- --> v1:172.21.15.134:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f170813bb60 con 0x7f17080a98f0 2024-03-25T17:40:49.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.719+0000 7f170daff640 10 monclient(hunting): _renew_subs 2024-03-25T17:40:49.722 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.719+0000 7f170daff640 10 monclient(hunting): authenticate will time out at 6002.981803s 2024-03-25T17:40:49.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.726+0000 7f1706ffd640 1 --1- >> v1:172.21.15.159:6789/0 conn(0x7f170813f710 0x7f170814bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.159:6789/0 says I am v1:172.21.15.134:43138/0 (socket says 172.21.15.134:43138) 2024-03-25T17:40:49.727 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.726+0000 7f1706ffd640 1 -- 172.21.15.134:0/132257357 learned_addr learned my addr 172.21.15.134:0/132257357 (peer_addr_for_me v1:172.21.15.134:0/0) 2024-03-25T17:40:49.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.729+0000 7f16effff640 10 client.?.objecter ms_handle_connect 0x7f170813f710 2024-03-25T17:40:49.730 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.729+0000 7f16effff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:49.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.732+0000 7f16effff640 10 client.?.objecter ms_handle_connect 0x7f17080a98f0 2024-03-25T17:40:49.732 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.732+0000 7f16effff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232935769 0 0) 0x7f170813b8f0 con 0x7f170813f710 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 10 monclient(hunting): my global_id is 4753 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.159:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f4004eb0 con 0x7f170813f710 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 10 client.?.objecter ms_handle_connect 0x7f17081430c0 2024-03-25T17:40:49.735 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.734+0000 7f16effff640 10 client.?.objecter resend_mon_ops 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.2 v1:172.21.15.134:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542569211 0 0) 0x7f170813bb60 con 0x7f17080a98f0 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): my global_id is 34706 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.134:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f170813b8f0 con 0x7f17080a98f0 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972410753 0 0) 0x7f16f4004eb0 con 0x7f170813f710 2024-03-25T17:40:49.744 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.159:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f170813bb60 con 0x7f170813f710 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f16fc001a10 con 0x7f170813f710 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): dump: 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:49.745 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.743+0000 7f16effff640 10 monclient(hunting): _finish_auth 0 2024-03-25T17:40:49.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.751+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.0 v1:172.21.15.134:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727642835 0 0) 0x7f170813b0f0 con 0x7f17081430c0 2024-03-25T17:40:49.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.751+0000 7f16effff640 10 monclient(hunting): _negotiate 2024-03-25T17:40:49.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.751+0000 7f16effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T17:40:49.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.751+0000 7f16effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T17:40:49.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.751+0000 7f16effff640 10 monclient(hunting): my global_id is 4725 2024-03-25T17:40:49.752 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.752+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.134:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16f4004eb0 con 0x7f17081430c0 2024-03-25T17:40:49.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1921041859 0 0) 0x7f170813bb60 con 0x7f170813f710 2024-03-25T17:40:49.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 >> v1:172.21.15.134:6790/0 conn(0x7f17080a98f0 legacy=0x7f1708139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:49.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 >> v1:172.21.15.134:6789/0 conn(0x7f17081430c0 legacy=0x7f17081e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:49.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 10 monclient: _finish_hunting 0 2024-03-25T17:40:49.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 1 monclient: found mon.b 2024-03-25T17:40:49.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T17:40:49.759 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.159:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170813be40 con 0x7f170813f710 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 10 monclient: _finish_auth 0 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 10 monclient: _check_auth_tickets 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f16effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 5 monclient: authenticate success, global_id 4753 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 10 monclient: _renew_subs 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.159:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170814e670 con 0x7f170813f710 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 10 client.4753.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 10 monclient: _renew_subs 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.159:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f170813ca00 con 0x7f170813f710 2024-03-25T17:40:49.760 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.758+0000 7f170daff640 1 librados: init done 2024-03-25T17:40:49.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.760+0000 7f170daff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:49.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.760+0000 7f170daff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:49.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.760+0000 7f170daff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:49.761 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:49.760+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17080627b0 con 0x7f170813f710 2024-03-25T17:40:50.042 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.042+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16fc001c70 con 0x7f170813f710 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.042+0000 7f16effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.042+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1610455452 0 0) 0x7f16fc003a60 con 0x7f170813f710 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.042+0000 7f16effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.042+0000 7f16effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.042+0000 7f16effff640 10 monclient: dump: 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:epoch 1 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:fsid d322d3f4-cbbc-46c7-9a8d-3205a340418d 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:last_changed 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:created 2024-03-25T17:28:22.040729+0000 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:min_mon_release 19 (squid) 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:election_strategy: 3 2024-03-25T17:40:50.043 INFO:teuthology.misc.health.smithi134.stderr:0: v1:172.21.15.134:6789/0 mon.a 2024-03-25T17:40:50.044 INFO:teuthology.misc.health.smithi134.stderr:1: v1:172.21.15.159:6789/0 mon.b 2024-03-25T17:40:50.044 INFO:teuthology.misc.health.smithi134.stderr:2: v1:172.21.15.134:6790/0 mon.c 2024-03-25T17:40:50.044 INFO:teuthology.misc.health.smithi134.stderr: 2024-03-25T17:40:50.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.042+0000 7f1705ffb640 4 set_mon_vals no callback set 2024-03-25T17:40:50.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.043+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2927627112 0 0) 0x7f16fc003c40 con 0x7f170813f710 2024-03-25T17:40:50.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.043+0000 7f17077fe640 10 monclient: get_auth_request con 0x7f16f406de00 auth_method 0 2024-03-25T17:40:50.044 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.043+0000 7f16effff640 10 client.4753.objecter ms_handle_connect 0x7f16f406de00 2024-03-25T17:40:50.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 3065053394 0 0) 0x7f16fc0548f0 con 0x7f170813f710 2024-03-25T17:40:50.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 10 client.4753.objecter ms_dispatch 0x7f1708000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T17:40:50.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 3 client.4753.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T17:40:50.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 3 client.4753.objecter handle_osd_map decoding full epoch 12 2024-03-25T17:40:50.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 20 client.4753.objecter dump_active .. 0 homeless 2024-03-25T17:40:50.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16fc0b18b0 con 0x7f170813f710 2024-03-25T17:40:50.048 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T17:40:50.049 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.047+0000 7f16effff640 10 monclient: _finish_command 1 = system:0 2024-03-25T17:40:50.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.307+0000 7f170daff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:50.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.307+0000 7f170daff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:50.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.307+0000 7f170daff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.159:6789/0 2024-03-25T17:40:50.308 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.307+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 --> v1:172.21.15.159:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17081398e0 con 0x7f170813f710 2024-03-25T17:40:50.342 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.342+0000 7f16effff640 1 -- 172.21.15.134:0/132257357 <== mon.1 v1:172.21.15.159:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 678709207) 0x7f16fc053fc0 con 0x7f170813f710 2024-03-25T17:40:50.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.342+0000 7f16effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T17:40:50.343 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.342+0000 7f16effff640 10 monclient: _finish_command 2 = system:0 2024-03-25T17:40:50.343 INFO:teuthology.misc.health.smithi134.stdout: 2024-03-25T17:40:50.343 INFO:teuthology.misc.health.smithi134.stdout:{"epoch":12,"fsid":"d322d3f4-cbbc-46c7-9a8d-3205a340418d","created":"2024-03-25T17:29:22.114783+0000","modified":"2024-03-25T17:39:30.927245+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T17:39:30.927245+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bee90ced-7048-4b40-a256-52b8e0df28b8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"78238efc-fb2f-47d4-a7f8-25baa1cbb58b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"3e547d19-49b0-4ad4-9b35-3c9702151be7","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"dc538434-60ca-4c73-9226-d24a8316d9a4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"48552c57-30eb-4b13-b468-0417c4e03ad4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"5f4c8c05-9231-4a4b-b76f-602253e33077","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"476003d3-e058-44fb-81a5-88ae20be2c3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"58eea41e-4a3f-4c11-86cb-a8d3d047599f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 10 librados: watch_flush enter 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 10 librados: watch_flush exit 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 20 client.4753.objecter shutdown clearing up homeless session... 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 10 client.4753.objecter successfully canceled tick 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 >> v1:172.21.15.134:6800/610837766 conn(0x7f16f406de00 legacy=0x7f16f40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 10 monclient: shutdown 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 >> v1:172.21.15.159:6789/0 conn(0x7f170813f710 legacy=0x7f170814bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 shutdown_connections 2024-03-25T17:40:50.346 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 >> 172.21.15.134:0/132257357 conn(0x7f17080b7ea0 msgr2=0x7f170814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T17:40:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 shutdown_connections 2024-03-25T17:40:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 1 -- 172.21.15.134:0/132257357 wait complete. 2024-03-25T17:40:50.347 INFO:teuthology.misc.health.smithi134.stderr:2024-03-25T17:40:50.345+0000 7f170daff640 1 librados: shutdown 2024-03-25T17:40:50.357 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T17:40:50.357 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T17:40:50.359 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T17:40:50.359 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T17:40:50.359 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:40:52.283 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:11:23.688 35535== 2024-03-25T17:40:52.284 INFO:tasks.ceph.osd.0.smithi134.stderr:==00:00:11:23.688 35535== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:40:52.318 INFO:tasks.ceph.osd.0.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:40:56.382 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:01.397 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.426 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:01.454 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.426 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:01.443 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.427 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:01.455 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.427 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:01.444 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.438 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:01.455 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.453 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:01.481 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.460 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:01.476 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:40:56.461 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T17:40:56.461 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T17:40:56.462 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T17:40:56.462 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:40:56.482 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:01.498 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:40:58.516 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:11:29.933 35528== 2024-03-25T17:40:58.516 INFO:tasks.ceph.osd.1.smithi134.stderr:==00:00:11:29.933 35528== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:40:58.560 INFO:tasks.ceph.osd.1.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:02.564 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T17:41:02.565 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T17:41:02.565 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T17:41:02.565 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:04.379 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:11:35.781 35537== 2024-03-25T17:41:04.379 INFO:tasks.ceph.osd.2.smithi134.stderr:==00:00:11:35.781 35537== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:04.419 INFO:tasks.ceph.osd.2.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:08.667 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T17:41:08.667 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T17:41:08.667 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T17:41:08.668 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:10.607 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:11:42.012 35536== 2024-03-25T17:41:10.608 INFO:tasks.ceph.osd.3.smithi134.stderr:==00:00:11:42.012 35536== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:10.651 INFO:tasks.ceph.osd.3.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:14.769 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:14.769 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T17:41:14.769 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T17:41:14.770 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T17:41:14.770 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:16.709 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:11:48.096 35440== 2024-03-25T17:41:16.709 INFO:tasks.ceph.osd.4.smithi159.stderr:==00:00:11:48.096 35440== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:16.744 INFO:tasks.ceph.osd.4.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:20.872 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:20.872 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T17:41:20.872 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T17:41:20.872 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T17:41:20.873 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:21.467 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:26.484 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.469 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:26.498 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.470 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:26.487 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.475 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:26.503 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.490 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:26.507 35189== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.495 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:26.524 34896== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.515 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:26.532 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.520 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:26.537 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:21.537 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:26.554 35192== Warning: unimplemented fcntl command: 1036 2024-03-25T17:41:22.824 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:11:54.207 35441== 2024-03-25T17:41:22.825 INFO:tasks.ceph.osd.5.smithi159.stderr:==00:00:11:54.207 35441== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:22.864 INFO:tasks.ceph.osd.5.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:26.974 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:26.974 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T17:41:26.974 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T17:41:26.975 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T17:41:26.975 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:28.829 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:12:00.212 35442== 2024-03-25T17:41:28.830 INFO:tasks.ceph.osd.6.smithi159.stderr:==00:00:12:00.212 35442== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:28.877 INFO:tasks.ceph.osd.6.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:33.076 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:33.076 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T17:41:33.077 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T17:41:33.077 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T17:41:33.077 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:34.502 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:12:05.877 35443== 2024-03-25T17:41:34.502 INFO:tasks.ceph.osd.7.smithi159.stderr:==00:00:12:05.877 35443== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:34.530 INFO:tasks.ceph.osd.7.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:39.179 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:39.179 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T17:41:39.180 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T17:41:39.180 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T17:41:39.180 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T17:41:39.180 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:39.181 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:41:39.181+0000 7f87bd766640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35198) UID: 0 2024-03-25T17:41:39.181 INFO:tasks.ceph.mgr.y.smithi134.stderr:2024-03-25T17:41:39.181+0000 7f87bd766640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T17:41:39.206 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T17:41:39.206 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T17:41:39.207 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:39.234 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T17:41:39.234 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T17:41:39.234 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T17:41:39.235 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:39.262 INFO:tasks.ceph.mon.a.smithi134.stderr:2024-03-25T17:41:39.250+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35158) UID: 0 2024-03-25T17:41:39.262 INFO:tasks.ceph.mon.a.smithi134.stderr:2024-03-25T17:41:39.252+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T17:41:39.921 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:44.938 35189== 2024-03-25T17:41:39.922 INFO:tasks.ceph.mon.a.smithi134.stderr:==00:00:12:44.938 35189== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:39.931 INFO:tasks.ceph.mon.a.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:45.336 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:45.336 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T17:41:45.337 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T17:41:45.337 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T17:41:45.337 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:45.357 INFO:tasks.ceph.mon.c.smithi134.stderr:2024-03-25T17:41:45.341+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35160) UID: 0 2024-03-25T17:41:45.358 INFO:tasks.ceph.mon.c.smithi134.stderr:2024-03-25T17:41:45.343+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T17:41:46.017 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:51.034 35192== 2024-03-25T17:41:46.018 INFO:tasks.ceph.mon.c.smithi134.stderr:==00:00:12:51.034 35192== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:46.026 INFO:tasks.ceph.mon.c.smithi134.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:51.439 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:51.440 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi134 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T17:41:51.440 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T17:41:51.440 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T17:41:51.440 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T17:41:51.456 INFO:tasks.ceph.mon.b.smithi159.stderr:2024-03-25T17:41:51.444+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34892) UID: 0 2024-03-25T17:41:51.457 INFO:tasks.ceph.mon.b.smithi159.stderr:2024-03-25T17:41:51.446+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T17:41:51.888 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:56.917 34896== 2024-03-25T17:41:51.888 INFO:tasks.ceph.mon.b.smithi159.stderr:==00:00:12:56.917 34896== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T17:41:51.898 INFO:tasks.ceph.mon.b.smithi159.stderr:daemon-helper: command failed with exit status 42 2024-03-25T17:41:57.542 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T17:41:57.542 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T17:41:57.543 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T17:41:57.543 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T17:41:57.543 DEBUG:teuthology.orchestra.run.smithi134:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T17:41:57.589 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:41:57.589 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T17:41:57.657 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:41:57.657 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T17:41:57.739 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:41:57.739 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T17:41:57.798 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:41:57.799 DEBUG:teuthology.orchestra.run.smithi134:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T17:41:57.857 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:41:57.857 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T17:41:58.012 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:41:58.012 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T17:41:58.084 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:41:58.084 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T17:41:58.141 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:41:58.141 DEBUG:teuthology.orchestra.run.smithi159:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T17:41:58.219 INFO:tasks.ceph:Archiving mon data... 2024-03-25T17:41:58.221 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/data/mon.a.tgz 2024-03-25T17:41:58.221 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-25T17:41:58.237 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.iBcmJJ1Acn 2024-03-25T17:41:58.237 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.iBcmJJ1Acn 2024-03-25T17:41:58.343 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.iBcmJJ1Acn 2024-03-25T17:41:58.429 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.iBcmJJ1Acn is 133KB 2024-03-25T17:41:58.481 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.iBcmJJ1Acn 2024-03-25T17:41:58.496 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/data/mon.c.tgz 2024-03-25T17:41:58.496 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-03-25T17:41:58.549 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.74XEQck1ir 2024-03-25T17:41:58.550 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.74XEQck1ir 2024-03-25T17:41:58.638 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.74XEQck1ir 2024-03-25T17:41:58.722 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.74XEQck1ir is 135KB 2024-03-25T17:41:58.773 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.74XEQck1ir 2024-03-25T17:41:58.787 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/data/mon.b.tgz 2024-03-25T17:41:58.787 DEBUG:teuthology.orchestra.run.smithi159:> mktemp 2024-03-25T17:41:58.802 INFO:teuthology.orchestra.run.smithi159.stdout:/tmp/tmp.DyR4so22Gu 2024-03-25T17:41:58.802 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DyR4so22Gu 2024-03-25T17:41:58.905 DEBUG:teuthology.orchestra.run.smithi159:> sudo chmod 0666 /tmp/tmp.DyR4so22Gu 2024-03-25T17:41:58.989 DEBUG:teuthology.orchestra.remote:smithi159:/tmp/tmp.DyR4so22Gu is 134KB 2024-03-25T17:41:59.038 DEBUG:teuthology.orchestra.run.smithi159:> rm -fr /tmp/tmp.DyR4so22Gu 2024-03-25T17:41:59.052 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T17:41:59.052 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T17:41:59.055 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T17:41:59.355 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:41:59.624 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T17:41:59.626 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/remote/smithi134/crash 2024-03-25T17:41:59.627 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T17:41:59.654 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/remote/smithi159/crash 2024-03-25T17:41:59.654 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T17:41:59.681 INFO:tasks.ceph:Compressing logs... 2024-03-25T17:41:59.681 DEBUG:teuthology.orchestra.run.smithi134:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T17:41:59.698 DEBUG:teuthology.orchestra.run.smithi159:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T17:41:59.721 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41462.log 2024-03-25T17:41:59.721 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T17:41:59.769 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T17:41:59.769 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35130.log 2024-03-25T17:41:59.769 INFO:teuthology.orchestra.run.smithi159.stderr:gzip/var/log/ceph/ceph-osd.6.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35210.log 2024-03-25T17:41:59.769 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35130.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T17:41:59.769 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35210.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T17:41:59.769 INFO:teuthology.orchestra.run.smithi159.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35130.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.7.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.35210.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35341.log 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35250.log 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr: 75.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35341.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35250.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35341.log.gz 2024-03-25T17:41:59.770 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35050.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35250.log.gz 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35301.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr: 88.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35050.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35301.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35050.log.gz 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35090.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.7.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35301.log.gz 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35090.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-mgr.x.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35090.log.gz 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35170.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr:/var/log/ceph/ceph-client.admin.35170.log: /var/log/ceph/ceph-mon.b.log: 87.7% 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T17:41:59.771 INFO:teuthology.orchestra.run.smithi159.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35170.log.gz 2024-03-25T17:41:59.772 INFO:teuthology.orchestra.run.smithi159.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T17:41:59.778 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2024-03-25T17:41:59.778 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40135.log 2024-03-25T17:41:59.778 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41462.log.gz 2024-03-25T17:41:59.778 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-03-25T17:41:59.778 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40135.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41032.log 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39471.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40135.log.gz 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35757.log 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41032.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39156.log 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41032.log.gz 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41071.log 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35757.log.gz 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39156.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41891.log 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41423.log 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39156.log.gz 2024-03-25T17:41:59.779 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41891.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41071.log.gz 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41423.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41891.log.gz 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41423.log.gz 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.0.log: /var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36640.log 2024-03-25T17:41:59.780 INFO:teuthology.orchestra.run.smithi134.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41189.log 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.36640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39115.log 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35200.log 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36640.log.gz 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41189.log.gz 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35200.log: gzip 86.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.39115.log.gz 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr: -- /var/log/ceph/ceph-client.admin.40369.log 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39628.log 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.35200.log.gz 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-03-25T17:41:59.781 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40876.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40369.log.gz 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39784.log 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39667.log 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39628.log.gz 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40174.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39784.log.gz 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40018.log 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35546.log 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39667.log.gz 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40525.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40018.log.gz 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41774.log 2024-03-25T17:41:59.782 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41618.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39354.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35546.log.gz 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41618.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41774.log.gz 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41696.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40486.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41618.log.gz 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41696.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr: 86.2%/var/log/ceph/ceph-client.admin.40486.log: -- replaced with /var/log/ceph/ceph-client.admin.41696.log.gz 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41384.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.audit.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40486.log.gz 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40954.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39706.log 2024-03-25T17:41:59.783 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40954.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41384.log.gz 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39745.log 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40954.log.gzgzip 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41501.log 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39706.log: /var/log/ceph/ceph-client.admin.39745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40447.log 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35608.log 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39745.log.gz 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39706.log.gz 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41735.log 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41501.log.gz 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.35608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42008.log 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40447.log.gz 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41306.log 2024-03-25T17:41:59.784 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35608.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40603.log 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41735.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37631.log 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42008.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40057.log 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41306.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.37631.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40603.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39589.log 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37631.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41267.log 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40057.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39589.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-03-25T17:41:59.785 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40252.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40837.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39589.log.gz 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42338.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41267.log.gz 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40837.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40252.log.gz 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40720.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40837.log.gz 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39511.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42338.log.gz 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40720.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38899.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39901.log 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40720.log.gz 2024-03-25T17:41:59.786 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.38899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39237.log 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39511.log.gz 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42377.log 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38899.log.gz 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39901.log.gz 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40798.log 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39237.log.gz 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40096.log 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr: 86.4%/var/log/ceph/ceph-client.admin.40798.log: -- replaced with /var/log/ceph/ceph-client.admin.42377.log.gz 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40681.log 86.4% 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40096.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40798.log.gz 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41228.log 2024-03-25T17:41:59.787 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/ceph-client.admin.40681.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40759.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40096.log.gz 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41228.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40681.log.gz 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39550.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41228.log.gz 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/ceph-osd.3.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41345.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40759.log.gz 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39550.log.gz 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- 86.4% /var/log/ceph/ceph-client.admin.40330.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41345.log.gz 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39315.log: /var/log/ceph/ceph-client.admin.40408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-03-25T17:41:59.788 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40330.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40330.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29348.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.tmp-client.admin.29348.log: gzip 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: -5 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29348.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40213.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42416.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42216.log 2024-03-25T17:41:59.789 INFO:teuthology.orchestra.run.smithi134.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40213.log.gz 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42047.log 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip/var/log/ceph/ceph-client.admin.42216.log: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42416.log.gz 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42299.log 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42216.log.gz 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr: --verbose/var/log/ceph/ceph-osd.0.log: -- /var/log/ceph/ceph-client.admin.42164.log 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42299.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42164.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42299.log.gz 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39979.log 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:gzip 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42164.log.gz 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40915.log 2024-03-25T17:41:59.790 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.39979.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40915.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39979.log.gz 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose 87.3% -- /var/log/ceph/ceph-client.admin.40564.log 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40915.log.gz 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-osd.2.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.40291.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40564.log.gz 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39823.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41930.log 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41930.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-03-25T17:41:59.791 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39940.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41930.log.gz 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39862.log 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41150.log 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39432.log: /var/log/ceph/ceph-client.admin.39862.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39862.log.gz 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42125.log 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42125.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41150.log.gz 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41657.log 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42125.log.gz 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.39198.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41540.log 2024-03-25T17:41:59.792 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41657.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T17:41:59.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.41540.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41657.log.gz 2024-03-25T17:41:59.793 INFO:teuthology.orchestra.run.smithi134.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42257.log 2024-03-25T17:41:59.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-mon.c.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41540.log.gz 2024-03-25T17:41:59.793 INFO:teuthology.orchestra.run.smithi134.stderr:/var/log/ceph/ceph-client.admin.42257.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42257.log.gz 2024-03-25T17:41:59.821 INFO:teuthology.orchestra.run.smithi134.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T17:42:00.562 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T17:42:00.582 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T17:42:00.625 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T17:42:00.636 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T17:42:00.869 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T17:42:00.892 INFO:teuthology.orchestra.run.smithi134.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T17:42:00.955 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T17:42:01.013 INFO:teuthology.orchestra.run.smithi159.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T17:42:04.858 INFO:teuthology.orchestra.run.smithi159.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T17:42:04.859 INFO:teuthology.orchestra.run.smithi159.stderr: 2024-03-25T17:42:04.860 INFO:teuthology.orchestra.run.smithi159.stderr:real 0m5.119s 2024-03-25T17:42:04.860 INFO:teuthology.orchestra.run.smithi159.stderr:user 0m8.988s 2024-03-25T17:42:04.860 INFO:teuthology.orchestra.run.smithi159.stderr:sys 0m0.355s 2024-03-25T17:42:04.890 INFO:teuthology.orchestra.run.smithi134.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T17:42:05.094 INFO:teuthology.orchestra.run.smithi134.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T17:42:05.097 INFO:teuthology.orchestra.run.smithi134.stderr: 2024-03-25T17:42:05.097 INFO:teuthology.orchestra.run.smithi134.stderr:real 0m5.388s 2024-03-25T17:42:05.097 INFO:teuthology.orchestra.run.smithi134.stderr:user 0m14.067s 2024-03-25T17:42:05.097 INFO:teuthology.orchestra.run.smithi134.stderr:sys 0m0.549s 2024-03-25T17:42:05.098 INFO:tasks.ceph:Archiving logs... 2024-03-25T17:42:05.098 DEBUG:teuthology.misc:Transferring archived files from smithi134:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/remote/smithi134/log 2024-03-25T17:42:05.099 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T17:42:07.549 DEBUG:teuthology.misc:Transferring archived files from smithi159:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/remote/smithi159/log 2024-03-25T17:42:07.549 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T17:42:09.118 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:42:09.325 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=65fbfccf37704aee96c46280e8152138 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T17:42:09.329 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T17:42:09.338 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T17:42:09.375 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T17:42:09.375 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T17:42:09.389 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T17:42:09.466 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T17:42:09.474 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T17:42:09.474 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:42:09.477 DEBUG:teuthology.orchestra.run.smithi159:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:42:09.490 INFO:teuthology.orchestra.run.smithi134.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:42:09.491 INFO:teuthology.orchestra.run.smithi159.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:42:09.495 INFO:teuthology.orchestra.run.smithi134.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:42:09.495 INFO:teuthology.orchestra.run.smithi134.stdout:=============================================================================== 2024-03-25T17:42:09.495 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 63 +653us[ +782us] +/- 73ms 2024-03-25T17:42:09.495 INFO:teuthology.orchestra.run.smithi134.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -1366us[-1235us] +/- 58ms 2024-03-25T17:42:09.495 INFO:teuthology.orchestra.run.smithi134.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 3 +995us[ +995us] +/- 62ms 2024-03-25T17:42:09.495 INFO:teuthology.orchestra.run.smithi134.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:42:09.497 INFO:teuthology.orchestra.run.smithi159.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:42:09.497 INFO:teuthology.orchestra.run.smithi159.stdout:=============================================================================== 2024-03-25T17:42:09.497 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +654us[ +654us] +/- 74ms 2024-03-25T17:42:09.497 INFO:teuthology.orchestra.run.smithi159.stdout:^* hv02.front.sepia.ceph.com 3 6 377 3 -1360us[-1264us] +/- 58ms 2024-03-25T17:42:09.497 INFO:teuthology.orchestra.run.smithi159.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 64 +1007us[+1103us] +/- 62ms 2024-03-25T17:42:09.497 INFO:teuthology.orchestra.run.smithi159.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:42:09.497 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T17:42:09.506 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T17:42:09.507 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T17:42:09.514 DEBUG:teuthology.orchestra.run.smithi134:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T17:42:09.608 DEBUG:teuthology.orchestra.run.smithi159:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T17:42:09.674 DEBUG:teuthology.orchestra.run.smithi134:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:42:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:42:09.704 DEBUG:teuthology.orchestra.run.smithi159:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:42:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:42:09.734 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T17:42:09.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T17:42:09.752 INFO:teuthology.task.internal:Duration was 5728.924952 seconds 2024-03-25T17:42:09.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T17:42:09.761 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T17:42:09.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T17:42:09.764 DEBUG:teuthology.orchestra.run.smithi159:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T17:42:09.805 INFO:teuthology.orchestra.run.smithi134.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:42:09.817 INFO:teuthology.orchestra.run.smithi159.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:42:10.071 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T17:42:10.071 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-03-25T17:42:10.071 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T17:42:10.092 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi159.front.sepia.ceph.com 2024-03-25T17:42:10.092 DEBUG:teuthology.orchestra.run.smithi159:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T17:42:10.191 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T17:42:10.192 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T17:42:10.194 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T17:42:10.224 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T17:42:10.224 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T17:42:10.263 DEBUG:teuthology.orchestra.run.smithi159:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T17:42:10.367 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T17:42:10.376 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T17:42:10.377 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T17:42:10.403 DEBUG:teuthology.orchestra.run.smithi159:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T17:42:10.430 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T17:42:10.440 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:42:10.446 DEBUG:teuthology.orchestra.run.smithi159:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:42:10.469 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-03-25T17:42:10.497 INFO:teuthology.orchestra.run.smithi159.stdout:kernel.core_pattern = core 2024-03-25T17:42:10.513 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:42:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:42:10.549 DEBUG:teuthology.orchestra.run.smithi159:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T17:42:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:42:10.568 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T17:42:10.577 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T17:42:10.578 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/remote/smithi134 2024-03-25T17:42:10.578 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T17:42:10.625 DEBUG:teuthology.misc:Transferring archived files from smithi159:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243/remote/smithi159 2024-03-25T17:42:10.625 DEBUG:teuthology.orchestra.run.smithi159:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T17:42:10.660 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T17:42:10.660 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T17:42:10.662 DEBUG:teuthology.orchestra.run.smithi159:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T17:42:10.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T17:42:10.720 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T17:42:10.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T17:42:10.729 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T17:42:10.730 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T17:42:10.732 DEBUG:teuthology.orchestra.run.smithi159:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T17:42:10.744 INFO:teuthology.orchestra.run.smithi134.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 17:42 /home/ubuntu/cephtest 2024-03-25T17:42:10.763 INFO:teuthology.orchestra.run.smithi159.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 17:42 /home/ubuntu/cephtest 2024-03-25T17:42:10.765 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T17:42:10.774 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T17:42:10.794 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T17:42:10.821 DEBUG:teuthology.nuke:shortname: smithi134 2024-03-25T17:42:10.821 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:42:10.843 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.840247', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpd+Wy2Iyel6QblPc3uH35qdQGkFJ3LR0wpRQxPwEiG4A/yQjQmul8p7bjNIlRnJ5eUtXz3mw7mevtg6b9gLDA='} 2024-03-25T17:42:10.846 DEBUG:teuthology.nuke:shortname: smithi159 2024-03-25T17:42:10.846 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:42:10.868 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi159.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621243', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 15:57:59.838679', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNoDZvHzI5Yx7J7zf/elHmBDSn0RqC4PyojPaeAHlb5s3UwoIlI+7Q97xPwEovjZNdacoFeMV9QOEgynzu+UpP4='} 2024-03-25T17:42:10.892 INFO:teuthology.orchestra.console.smithi134:Power off 2024-03-25T17:42:10.893 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T17:42:10.916 DEBUG:teuthology.orchestra.console.smithi134:power off output: Chassis Power Control: Down/Off 2024-03-25T17:42:10.916 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:10.932 INFO:teuthology.orchestra.console.smithi159:Power off 2024-03-25T17:42:10.932 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T17:42:10.942 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-25T17:42:10.955 DEBUG:teuthology.orchestra.console.smithi159:power off output: Chassis Power Control: Down/Off 2024-03-25T17:42:10.955 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:10.977 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-25T17:42:14.943 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:14.978 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:15.053 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is on 2024-03-25T17:42:15.089 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-25T17:42:19.054 DEBUG:teuthology.orchestra.console.smithi134:pexpect command: ipmitool -H smithi134.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:19.090 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:19.166 DEBUG:teuthology.orchestra.console.smithi134:check power output: Chassis Power is off 2024-03-25T17:42:19.199 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is on 2024-03-25T17:42:19.267 INFO:teuthology.orchestra.console.smithi134:Power off completed 2024-03-25T17:42:23.203 DEBUG:teuthology.orchestra.console.smithi159:pexpect command: ipmitool -H smithi159.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T17:42:23.313 DEBUG:teuthology.orchestra.console.smithi159:check power output: Chassis Power is off 2024-03-25T17:42:23.414 INFO:teuthology.orchestra.console.smithi159:Power off completed 2024-03-25T17:42:23.519 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 5728.924951791763 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=65fbfccf37704aee96c46280e8152138 status: fail success: false 2024-03-25T17:42:23.519 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:42:23.591 INFO:teuthology.run:FAIL